IP-Adapter

___*IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models*___

Introduction

we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to-image diffusion models. An IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fine-tuned image prompt model. IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. Moreover, the image prompt can also work well with the text prompt to accomplish multimodal image generation.

Release
- [2024/01/19] 🔥 Add IP-Adapter-FaceID-Portrait, more information can be found here.
- [2024/01/17] 🔥 Add an experimental version of IP-Adapter-FaceID-PlusV2 for SDXL, more information can be found here.
- [2024/01/04] 🔥 Add an experimental version of IP-Adapter-FaceID for SDXL, more information can be found here.
- [2023/12/29] 🔥 Add an experimental version of IP-Adapter-FaceID-PlusV2, more information can be found here.
- [2023/12/27] 🔥 Add an experimental version of IP-Adapter-FaceID-Plus, more information can be found here.
- [2023/12/20] 🔥 Add an experimental version of IP-Adapter-FaceID, more information can be found here.
- [2023/11/22] IP-Adapter is available in Diffusers thanks to Diffusers Team.
- [2023/11/10] 🔥 Add an updated version of IP-Adapter-Face. The demo is here.
- [2023/11/05] 🔥 Add text-to-image demo with IP-Adapter and Kandinsky 2.2 Prior
- [2023/11/02] Support safetensors
- [2023/9/08] 🔥 Update a new version of IP-Adapter with SDXL_1.0. More information can be found here.
- [2023/9/05] 🔥🔥🔥 IP-Adapter is supported in WebUI and ComfyUI (or ComfyUI_IPAdapter_plus).
- [2023/8/30] 🔥 Add an IP-Adapter with face image as prompt. The demo is here.
- [2023/8/29] 🔥 Release the training code.
- [2023/8/23] 🔥 Add code and models of IP-Adapter with fine-grained features. The demo is here.
- [2023/8/18] 🔥 Add code and models for SDXL 1.0. The demo is here.
- [2023/8/16] 🔥 We release the code and models.

Installation

install latest diffusers
pip install diffusers==0.22.1

install ip-adapter
pip install git+https://github.com/tencent-ailab/IP-Adapter.git

download the models
cd IP-Adapter
git lfs install
git clone https://huggingface.co/h94/IP-Adapter
mv IP-Adapter/models models
mv IP-Adapter/sdxl_models sdxl_models

then you can use the notebook

Download Models

you can download models from here. To run the demo, you should also download the following models:
- runwayml/stable-diffusion-v1-5
- stabilityai/sd-vae-ft-mse
- SG161222/Realistic_Vision_V4.0_noVAE
- ControlNet models

How to Use

SD_1.5

- ip_adapter_demo: image variations, image-to-image, and inpainting with image prompt.

- ip_adapter_controlnet_demo, ip_adapter_t2i-adapter: structural generation with image prompt.

- ip_adapter_multimodal_prompts_demo: generation with multimodal prompts.

- ip_adapter-plus_demo: the demo of IP-Adapter with fine-grained features.

- ip_adapter-plus-face_demo: generation with face image as prompt.

Best Practice
- If you only use the image prompt, you can set the sc

……（后略，见原文）