MotionCtrl

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

[SIGGRAPH 2024 CONFERENCE PROCEEDINGS]

👉 MotionCtrl for [SVD], for [VideoCrafter], for [AnimateDiff]

      

🤗    🤗 

🔥🔥🔥 We have released both Training and Inference code of MotionCtrl deployed on AnimateDiff

🔥🔥 We release the codes, models and demos for MotionCtrl on Stable Video Diffusion (SVD).

https://github.com/TencentARC/MotionCtrl/assets/19488619/45d44bf5-d4bf-4e45-8628-2c8926b5954a

Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.

Results of MotionCtrl+AnimateDiff

Results of MotionCtrl+SVD

More results are in showcase_svd and our Project Page.

Results of MotionCtrl+VideoCrafter

More results are in our Project Page.

📝 Changelog

- [x] 20231225: Release MotionCtrl deployed on *LVDM/VideoCrafter*.
- [x] 20231225: Gradio demo available. 🤗 
- [x] 20231228: Provide local gradio demo for convenience.
- [x] 20240115 More camera poses used for testing are provided in dataset/camera_poses
- [x] 20240115 Release MotionCtrl deployed on *SVD*. Codes are in the branch svd and Gradio Demo is available in 🤗 .
- [x] ❗❗❗ Gradio demo of MotionCtrl deployed on *VideoCrafter2* is available in 🤗 . You can also run it locally by python -m app --share.

- [x] ❗❗❗ Release MotionCtrl deployed on *AnimateDiff* are available in branch animatediff, containing both training and inference code.
- [x] 20240920 Provide scripts for collecting object trajectories with ParticleSfM. 
- [x] 20240920 We provide a HandyTrajDrawer to customize object trajectories more conveniently.

⚙️ Environment
    conda create -n motionctrl python=3.10.6
    conda activate motionctrl
    pip install -r requirements.txt

💫 Inference

- #### Run local inference script

1. Download the weights of MotionCtrl motionctrl.pth and put it to ./checkpoints.
2. Go into configs/inference/run.sh and set condtype as 'camera_motion', 'object_motion', or 'both'.
- condtype=camera_motion means only control the camera motion in the generated video.
- condtype=object_motion means only control the object motion in the generated video.
- condtype=both means control the camera motion and object motion in the generated video simultaneously.
3. Running scripts:
        sh configs/inference/run.sh

- #### Run local gradio demo
      python -m app --share

🔥🔥 Training 👉 Details

Preparing Dataset
  - RealEstate10K

    1.  Following https://github.com/cashiwamochi/RealEstate10K_Downloader to download and process the videos.
    2. Corresponding Captions and List are provided in GoogleDrive.

  - WebVid with Object Trajectories
    1. Preparing ParticleSfM. Our experiments is running on CentOS 8.5 and we provide a detailed install note in dataset/object_trajectories/ParticleSfM_Install_Note.pdf.
    2. Moving dataset/object_trajectories/prepare_webvideo_len32.py and dataset/object_trajectories/run_particlesfm_obj_traj.py to ParticleSfM project.
    3. Step 1: Prepare sub-videos with lenth of 32 and size of 256 x 256.

……（后略，见原文）