许可证: mit
许可证链接: https://huggingface.co/microsoft/Florence-2-large-ft/resolve/main/LICENSE
任务类型: 图像文本到文本
标签:

• 视觉

Florence-2：推进统一表示法应对多样化视觉任务

模型概述

此Hub仓库包含微软Florence-2模型的HuggingFace transformers实现。

Florence-2是一种先进的视觉基础模型，采用基于提示的方法处理广泛的视觉和视觉语言任务。它能通过简单文本提示执行图像描述、目标检测和分割等任务。该模型利用FLD-5B数据集（涵盖1.26亿张图像的54亿条标注）实现多任务学习。其序列到序列架构使其在零样本和微调场景中均表现优异，成为具有竞争力的视觉基础模型。

资源与技术文档：

• Florence-2技术报告
• Florence-2-large模型推理与可视化Jupyter笔记本

快速开始

以下代码展示如何使用模型（所有模型均采用float16训练）：

import requests
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)

prompt = "<OD>"
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    do_sample=False,
    num_beams=3
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))
print(parsed_answer)

任务能力

通过切换提示词，该模型可执行不同任务。首先定义任务执行函数：

import requests
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))
    print(parsed_answer)

支持任务示例

run_example("<CAPTION>")

run_example("<DETAILED_CAPTION>")

run_example("<MORE_DETAILED_CAPTION>")

run_example("<CAPTION_TO_PHRASE_GROUNDING>", "A green car parked in front of a yellow building.")

run_example("<OD>")

run_example("<DENSE_REGION_CAPTION>")

run_example("<REGION_PROPOSAL>")

run_example("<OCR>")

run_example("<OCR_WITH_REGION>")

性能基准

零样本性能

下表展示通用视觉基础模型在图像描述和目标检测任务上的零样本表现（未使用评估任务训练数据）：

视觉语言任务表现对比：

微调后性能

下表比较专家模型与通用模型在描述和VQA任务的表现（"▲"表示使用外部OCR输入）：

参照任务表现对比：

引用信息

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}