license: mit license_link: https://huggingface.co/microsoft/Florence-2-large/resolve/main/LICENSE pipeline_tag: image-text-to-text tags:

• vision

Florence-2（无flash-attn版）：推进多视觉任务的统一表征

⚠️ 此版本为Florence-2的修改版，通过劫持flash-attn方法并替换为常规注意力机制，移除了对安装flash-attn包的依赖。这可能会影响模型性能。

模型概述

本Hub仓库包含微软Florence-2模型的HuggingFace transformers实现。

Florence-2是一种先进的视觉基础模型，采用基于提示的方法处理广泛的视觉和视觉语言任务。它能通过简单文本提示执行图像描述、目标检测和分割等任务。该模型利用包含126万张图像、54亿标注的FLD-5B数据集进行多任务学习，其序列到序列架构使其在零样本和微调场景中均表现优异，成为极具竞争力的视觉基础模型。

技术资源：

• Florence-2技术报告
• Florence-2-large推理可视化Jupyter Notebook

快速开始

以下代码展示模型使用方法（所有模型均采用float16训练）：

import requests
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

prompt = "<OD>"  # 目标检测任务提示

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    num_beams=3,
    do_sample=False
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))
print(parsed_answer)

多任务支持

通过切换提示词，本模型可执行不同任务。首先定义任务执行函数：

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))
    print(parsed_answer)

支持的任务包括：

run_example("<CAPTION>")

run_example("<DETAILED_CAPTION>")

run_example("<MORE_DETAILED_CAPTION>")

run_example("<CAPTION_TO_PHRASE_GROUNDING>", "A green car parked in front of a yellow building.")

run_example("<OD>")

run_example("<DENSE_REGION_CAPTION>")

run_example("<REGION_PROPOSAL>")

run_example("<OCR>")

run_example("<OCR_WITH_REGION>")

性能基准

零样本性能

微调后性能

引用信息

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}