🚀 BRIA 2.3 ControlNet Generative Fill Fast

BRIA 2.3 Generative Fill是专门在最大的多源商业级许可数据集上进行训练的，在保证最佳质量的同时，可安全用于商业用途。该模型为版权和隐私侵权以及有害内容缓解提供了全面的法律责任保障，因为我们的数据集不包含受版权保护的材料，如虚构角色、标志或商标、公众人物、有害内容或侵犯隐私的内容。

此模型旨在根据用户提供的文本提示填充图像中的遮罩区域，可应用于不同场景，包括图像内的对象替换、添加和修改等。

该模型适用于所有类型的遮罩，但针对占据图像面积超过15%的Blob形状遮罩进行了高度优化。

🚀 快速开始

获取访问权限

BRIA 2.3 ControlNet - Generative Fill需要访问BRIA 2.3基础模型。更多信息，请点击此处。

• API端点：Bria.ai，fal.ai
• ComfyUI：在工作流中使用

更多信息，请访问我们的网站。

加入我们的Discord社区，获取更多信息、教程、工具，并与其他用户交流！

点击此处查看演示

✨ 主要特性

BRIA 2.3 ControlNet Generative Fill可以应用于BRIA 2.3文本到图像模型之上，因此能够使用Fast - LORA。这使得生成填充模型速度极快，在A10 GPU上仅需1.6秒。

📚 详细文档

模型描述

如何使用

测试环境如下：

diffusers==0.27.2
transformers==4.47.1
torch==2.3.0 (on CUDA 12.1)
peft==0.14.0
huggingface_hub==0.25.2

from diffusers import (
    AutoencoderKL,
    LCMScheduler,
)
from pipeline_controlnet_sd_xl import StableDiffusionXLControlNetPipeline
from controlnet import ControlNetModel, ControlNetConditioningEmbedding
import torch
import numpy as np
from PIL import Image
import requests
import PIL
from io import BytesIO
from torchvision import transforms
import pandas as pd 
import os 


def resize_image_to_retain_ratio(image):
    pixel_number = 1024*1024
    granularity_val = 8
    ratio = image.size[0] / image.size[1]
    width = int((pixel_number * ratio) ** 0.5)
    width = width - (width % granularity_val)
    height = int(pixel_number / width)
    height = height - (height % granularity_val)

    image = image.resize((width, height))
    return image


def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")


def get_masked_image(image, image_mask, width, height):
    image_mask = image_mask # fill area is white
    image_mask = image_mask.resize((width, height)) # object to remove is white (1)
    image_mask_pil = image_mask
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask_pil.convert("L")).astype(np.float32) / 255.0
    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    masked_image_to_present = image.copy()
    masked_image_to_present[image_mask > 0.5] = (0.5,0.5,0.5)  # set as masked pixel
    image[image_mask > 0.5] = 0.5  # set as masked pixel - s.t. will be grey 
    image = Image.fromarray((image * 255.0).astype(np.uint8))
    masked_image_to_present = Image.fromarray((masked_image_to_present * 255.0).astype(np.uint8))
    return image, image_mask_pil, masked_image_to_present


image_transforms = transforms.Compose(
    [
        transforms.ToTensor(),
    ]
)

default_negative_prompt = "blurry"

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url).resize((1024, 1024))
mask_image = download_image(mask_url).resize((1024, 1024))


init_image = resize_image_to_retain_ratio(init_image)
width, height = init_image.size

mask_image = mask_image.convert("L").resize(init_image.size)

width, height = init_image.size

# Load, init model    
controlnet = ControlNetModel().from_pretrained("briaai/BRIA-2.3-ControlNet-Generative-Fill", torch_dtype=torch.float16)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("briaai/BRIA-2.3", controlnet=controlnet.to(dtype=torch.float16), torch_dtype=torch.float16, vae=vae) #force_zeros_for_empty_prompt=False, # vae=vae)

pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights("briaai/BRIA-2.3-FAST-LORA")
pipe.fuse_lora()
pipe = pipe.to(device="cuda")

# pipe.enable_xformers_memory_efficient_attention()

generator = torch.Generator(device="cuda").manual_seed(123456)

vae = pipe.vae


masked_image, image_mask, masked_image_to_present = get_masked_image(init_image, mask_image, width, height)

masked_image_tensor = image_transforms(masked_image)
masked_image_tensor = (masked_image_tensor - 0.5) / 0.5


masked_image_tensor = masked_image_tensor.unsqueeze(0).to(device="cuda")
control_latents = vae.encode(  
        masked_image_tensor[:, :3, :, :].to(vae.dtype)
    ).latent_dist.sample()   
control_latents = control_latents * vae.config.scaling_factor 


image_mask = np.array(image_mask)[:,:]
mask_tensor = torch.tensor(image_mask, dtype=torch.float32)[None, ...]
# binarize the mask
mask_tensor = torch.where(mask_tensor > 128.0, 255.0, 0)       

mask_tensor = mask_tensor / 255.0

mask_tensor = mask_tensor.to(device="cuda")
mask_resized = torch.nn.functional.interpolate(mask_tensor[None, ...], size=(control_latents.shape[2], control_latents.shape[3]), mode='nearest')

masked_image = torch.cat([control_latents, mask_resized], dim=1)

prompt = ""

gen_img = pipe(negative_prompt=default_negative_prompt, prompt=prompt, 
            controlnet_conditioning_scale=1.0, 
            num_inference_steps=12, 
            height=height, width=width, 
            image = masked_image, # control image
            init_image = init_image,     
            mask_image = mask_tensor,
            guidance_scale = 1.2,
            generator=generator).images[0]

📄 许可证

• 许可证名称：bria - 2.3
• 许可证类型：其他
• 许可证链接：https://bria.ai/bria-huggingface-model-license-agreement/

模型权重可通过购买商业许可证获得。请填写以下表格，我们会与您联系。

• 模型商业许可申请：填写此表格以申请该模型的商业许可证。
  • 姓名：文本输入
  • 公司/组织名称：文本输入
  • 组织类型（早期/成长型初创企业、企业、学术机构）：文本输入
  • 职位：文本输入
  • 国家：文本输入
  • 电子邮件：文本输入
  • 勾选此项：表示我同意BRIA的隐私政策和条款条件，详见以下链接。