library_name: diffusers
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:

• lora
• text-to-image
  license: openrail++
  inference: false

潜在一致性模型(LCM) LoRA：SDXL版

潜在一致性模型(LCM) LoRA由Simian Luo、Yiqin Tan、Suraj Patil、Daniel Gu等在论文《LCM-LoRA：通用稳定扩散加速模块》中提出。

这是为stable-diffusion-xl-base-1.0设计的蒸馏一致性适配器，可将推理步骤缩减至仅需2-8步。

使用方法

LCM-LoRA从v0.23.0版本开始获得🤗 Hugging Face Diffusers库支持。运行前请先安装最新版Diffusers库及peft、accelerate、transformers：

pip install --upgrade pip  
pip install --upgrade diffusers transformers accelerate peft  

注：完整用法示例请参阅官方LCM-LoRA文档

文生图

适配器需与基础模型stabilityai/stable-diffusion-xl-base-1.0配合使用。需将调度器改为LCMScheduler，并将推理步骤设为2-8步。注意需禁用guidance_scale或将其值设为1.0-2.0之间。

import torch  
from diffusers import LCMScheduler, AutoPipelineForText2Image  

model_id = "stabilityai/stable-diffusion-xl-base-1.0"  
adapter_id = "latent-consistency/lcm-lora-sdxl"  

pipe = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16")  
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)  
pipe.to("cuda")  

# 加载并融合LCM LoRA  
pipe.load_lora_weights(adapter_id)  
pipe.fuse_lora()  

prompt = "自画像油画，一位金发机械美人，8K画质"  

# 通过传入0禁用guidance_scale  
image = pipe(prompt=prompt, num_inference_steps=4, guidance_scale=0).images[0]  

图像修复

LCM-LoRA同样支持修复任务：

import torch  
from diffusers import AutoPipelineForInpainting, LCMScheduler  
from diffusers.utils import load_image, make_image_grid  

pipe = AutoPipelineForInpainting.from_pretrained(  
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",  
    torch_dtype=torch.float16,  
    variant="fp16",  
).to("cuda")  

# 设置调度器  
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)  

# 加载LCM-LoRA  
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")  
pipe.fuse_lora()  

# 加载底图与掩膜  
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png").resize((1024, 1024))  
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png").resize((1024, 1024))  

prompt = "山顶城堡，精细刻画，8K画质"  
generator = torch.manual_seed(42)  
image = pipe(  
    prompt=prompt,  
    image=init_image,  
    mask_image=mask_image,  
    generator=generator,  
    num_inference_steps=5,  
    guidance_scale=4,  
).images[0]  
make_image_grid([init_image, mask_image, image], rows=1, cols=3)  

结合风格化LoRA

LCM-LoRA可与其他LoRA结合实现少步数（4-8步）风格化生成。以下示例将LCM-LoRA与剪纸风格LoRA结合使用。多LoRA组合方法详见本指南。

import torch  
from diffusers import DiffusionPipeline, LCMScheduler  

pipe = DiffusionPipeline.from_pretrained(  
    "stabilityai/stable-diffusion-xl-base-1.0",  
    variant="fp16",  
    torch_dtype=torch.float16  
).to("cuda")  

# 设置调度器  
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)  

# 加载双LoRA  
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl", adapter_name="lcm")  
pipe.load_lora_weights("TheLastBen/Papercut_SDXL", weight_name="papercut.safetensors", adapter_name="papercut")  

# 组合LoRA  
pipe.set_adapters(["lcm", "papercut"], adapter_weights=[1.0, 0.8])  

prompt = "剪纸风格，可爱狐狸"  
generator = torch.manual_seed(0)  
image = pipe(prompt, num_inference_steps=4, guidance_scale=1, generator=generator).images[0]  
image  

ControlNet控制生成

import torch  
import cv2  
import numpy as np  
from PIL import Image  

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, LCMScheduler  
from diffusers.utils import load_image  

image = load_image(  
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"  
).resize((1024, 1024))  

image = np.array(image)  

low_threshold = 100  
high_threshold = 200  

image = cv2.Canny(image, low_threshold, high_threshold)  
image = image[:, :, None]  
image = np.concatenate([image, image, image], axis=2)  
canny_image = Image.fromarray(image)  

controlnet = ControlNetModel.from_pretrained("diffusers/controlnet-canny-sdxl-1.0-small", torch_dtype=torch.float16, variant="fp16")  
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(  
    "stabilityai/stable-diffusion-xl-base-1.0",  
    controlnet=controlnet,  
    torch_dtype=torch.float16,  
    safety_checker=None,  
    variant="fp16"  
).to("cuda")  

# 设置调度器  
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)  

# 加载LCM-LoRA  
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")  
pipe.fuse_lora()  

generator = torch.manual_seed(0)  
image = pipe(  
    "蒙娜丽莎画像",  
    image=canny_image,  
    num_inference_steps=5,  
    guidance_scale=1.5,  
    controlnet_conditioning_scale=0.5,  
    cross_attention_kwargs={"scale": 1},  
    generator=generator,  
).images[0]  
make_image_grid([canny_image, image], rows=1, cols=2)  

<提示>
示例参数可能不适用于所有场景，建议尝试调整num_inference_steps、guidance_scale、controlnet_conditioning_scale和cross_attention_kwargs等参数。
</提示>

T2I适配器

本例展示LCM-LoRA与Canny T2I-Adapter及SDXL的配合使用：

import torch  
import cv2  
import numpy as np  
from PIL import Image  

from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, LCMScheduler  
from diffusers.utils import load_image, make_image_grid  

# 预处理图像（低分辨率检测边缘以避免高频细节）  
image = load_image(  
    "https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_canny.jpg"  
).resize((384, 384))  

image = np.array(image)  

low_threshold = 100  
high_threshold = 200  

image = cv2.Canny(image, low_threshold, high_threshold)  
image = image[:, :, None]  
image = np.concatenate([image, image, image], axis=2)  
canny_image = Image.fromarray(image).resize((1024, 1024))  

# 加载适配器  
adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-canny-sdxl-1.0", torch_dtype=torch.float16, varient="fp16").to("cuda")  

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(  
    "stabilityai/stable-diffusion-xl-base-1.0",   
    adapter=adapter,  
    torch_dtype=torch.float16,  
    variant="fp16",   
).to("cuda")  

# 设置调度器  
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)  

# 加载LCM-LoRA  
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")  

prompt = "真实系魔法精灵，4K高清画质"  
negative_prompt = "多余数字|缺数字|裁剪|低质量|画面故障|畸形|丑陋|扭曲"  

generator = torch.manual_seed(0)  
image = pipe(  
    prompt=prompt,  
    negative_prompt=negative_prompt,  
    image=canny_image,  
    num_inference_steps=4,  
    guidance_scale=1.5,   
    adapter_conditioning_scale=0.8,   
    adapter_conditioning_factor=1,  
    generator=generator,  
).images[0]  
make_image_grid([canny_image, image], rows=1, cols=2)  

速度基准测试

待补充

训练方案

待补充