🚀 BRIA 2.3 ControlNet Inpainting Fast

BRIA 2.3是一款专为商业用途打造的图像修复模型，它基于大规模多源商业级许可数据集进行训练，在保证图像质量的同时，为商业使用提供了安全保障。该模型能有效避免版权和隐私侵权问题，以及有害内容的产生。它可以根据用户提供的文本提示，填充图像中的掩码区域，适用于图像中物体的移除、替换、添加和修改等多种场景，还具备图像扩展能力。

🚀 快速开始

加入我们的 Discord社区，获取更多信息、教程和工具，还能与其他用户交流！

✨ 主要特性

• 全新特性：BRIA 2.3 ControlNet Inpainting 可以基于 BRIA 2.3 Text-to-Image 使用 Fast-LORA，实现极快的图像修复，在 A10 GPU 上仅需 1.6 秒。
• 高质量与安全性：仅在最大的多源商业级许可数据集上进行训练，保证最佳质量，同时确保商业使用安全。
• 法律责任保障：为版权和隐私侵权以及有害内容缓解提供全面的法律责任覆盖。
• 广泛应用场景：可用于图像中物体的移除、替换、添加、修改以及图像扩展等。

📦 安装指南

下载

from huggingface_hub import hf_hub_download
import os

try:
    local_dir = os.path.dirname(__file__)
except:
    local_dir = '.'

hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='controlnet.py', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='config.json', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='image_processor.py', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='pipeline_controlnet_sd_xl.py', local_dir=local_dir)

💻 使用示例

基础用法

from diffusers import (
    AutoencoderKL,
    LCMScheduler,
)
from pipeline_controlnet_sd_xl import StableDiffusionXLControlNetPipeline
from controlnet import ControlNetModel
import torch
import numpy as np
from PIL import Image
import requests
import PIL
from io import BytesIO
from torchvision import transforms
import os

def resize_image_to_retain_ratio(image):
    pixel_number = 1024*1024
    granularity_val = 8
    ratio = image.size[0] / image.size[1]
    width = int((pixel_number * ratio) ** 0.5)
    width = width - (width % granularity_val)
    height = int(pixel_number / width)
    height = height - (height % granularity_val)

    image = image.resize((width, height))
    return image

def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")

def get_masked_image(image, image_mask, width, height):
    image_mask = image_mask # inpaint area is white
    image_mask = image_mask.resize((width, height)) # object to remove is white (1)
    image_mask_pil = image_mask
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask_pil.convert("L")).astype(np.float32) / 255.0
    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    masked_image_to_present = image.copy()
    masked_image_to_present[image_mask > 0.5] = (0.5,0.5,0.5)  # set as masked pixel
    image[image_mask > 0.5] = 0.5  # set as masked pixel - s.t. will be grey 
    image = Image.fromarray((image * 255.0).astype(np.uint8))
    masked_image_to_present = Image.fromarray((masked_image_to_present * 255.0).astype(np.uint8))
    return image, image_mask_pil, masked_image_to_present

image_transforms = transforms.Compose(
    [
        transforms.ToTensor(),
    ]
)

default_negative_prompt = "Logo,Watermark,Text,Ugly,Morbid,Extra fingers,Poorly drawn hands,Mutation,Blurry,Extra limbs,Gross proportions,Missing arms,Mutated hands,Long neck,Duplicate,Mutilated,Mutilated hands,Poorly drawn face,Deformed,Bad anatomy,Cloned face,Malformed limbs,Missing legs,Too many fingers"

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url).resize((1024, 1024))
mask_image = download_image(mask_url).resize((1024, 1024))

init_image = resize_image_to_retain_ratio(init_image)
width, height = init_image.size

mask_image = mask_image.convert("L").resize(init_image.size)

width, height = init_image.size

# Load, init model    
controlnet = ControlNetModel().from_pretrained("briaai/BRIA-2.3-ControlNet-Inpainting", torch_dtype=torch.float16)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("briaai/BRIA-2.3", controlnet=controlnet.to(dtype=torch.float16), torch_dtype=torch.float16, vae=vae) #force_zeros_for_empty_prompt=False, # vae=vae)

pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights("briaai/BRIA-2.3-FAST-LORA")
pipe.fuse_lora()
pipe = pipe.to(device="cuda")

# pipe.enable_xformers_memory_efficient_attention()

generator = torch.Generator(device="cuda").manual_seed(123456)

vae = pipe.vae

masked_image, image_mask, masked_image_to_present = get_masked_image(init_image, mask_image, width, height)

masked_image_tensor = image_transforms(masked_image)
masked_image_tensor = (masked_image_tensor - 0.5) / 0.5

masked_image_tensor = masked_image_tensor.unsqueeze(0).to(device="cuda")
control_latents = vae.encode(  
        masked_image_tensor[:, :3, :, :].to(vae.dtype)
    ).latent_dist.sample()   
control_latents = control_latents * vae.config.scaling_factor 

image_mask = np.array(image_mask)[:,:]
mask_tensor = torch.tensor(image_mask, dtype=torch.float32)[None, ...]
# binarize the mask
mask_tensor = torch.where(mask_tensor > 128.0, 255.0, 0)       

mask_tensor = mask_tensor / 255.0

mask_tensor = mask_tensor.to(device="cuda")
mask_resized = torch.nn.functional.interpolate(mask_tensor[None, ...], size=(control_latents.shape[2], control_latents.shape[3]), mode='nearest')

masked_image = torch.cat([control_latents, mask_resized], dim=1)

prompt = ""

gen_img = pipe(negative_prompt=default_negative_prompt, prompt=prompt, 
            controlnet_conditioning_scale=1.0, 
            num_inference_steps=12, 
            height=height, width=width, 
            image = masked_image, # control image
            init_image = init_image,     
            mask_image = mask_tensor,
            guidance_scale = 1.2,
            generator=generator).images[0]

display(gen_img)

📚 详细文档

模型描述

获取源代码和预训练模型

如果您对 BRIA 2.3 修复模型感兴趣，可以购买使用许可。购买 BRIA 2.3 修复模型的访问权限可确保版税管理和商业使用的完全责任。

如果您是初创企业或学生，我们鼓励您申请我们的专业学术和 初创企业计划 以获取访问权限。这些计划旨在通过我们的前沿技术支持新兴企业和学术研究。

通过提交上述表格，您同意 BRIA 的 隐私政策 和 条款与条件。

📄 许可证

本模型使用 bria-2.3 inpainting Licensing terms & conditions 许可证。模型权重需购买商业许可证才能获取。请填写以下表格，我们会与您联系。