Kosmos 2 Patch14 224

由 microsoft 开发

Kosmos-2是一个多模态大语言模型，能够理解和生成与图像相关的文本描述，并实现文本与图像区域的关联。

图像生成文本

Transformers

开源协议:MIT #多模态锚定 #视觉语言理解 #指代表达生成

下载量 171.99k

发布时间 : 10/2/2023

模型介绍

内容详情

替代品

模型简介

Kosmos-2是一个视觉-语言模型，专注于图像描述生成和视觉接地任务。它能够理解图像内容并生成相关文本描述，同时还能将文本中的短语与图像中的特定区域关联起来。

模型特点

多模态接地能力

能够将文本中的短语与图像中的特定区域关联起来，实现精准的视觉定位

多模态指代理解

可以理解图像中的指代表达，并能生成描述图像区域的指代表达

多功能视觉-语言任务

支持多种视觉-语言任务，包括接地视觉问答、图像描述生成等

模型能力

图像描述生成

视觉接地

多模态指代理解

接地视觉问答

指代表达生成

使用案例

内容理解与生成

自动图像标注

为图像生成详细的文字描述

生成包含图像中主要对象和场景的描述文本

视觉问答系统

回答关于图像内容的特定问题

准确回答关于图像中对象位置和关系的问题

辅助技术

视觉辅助工具

为视障人士描述图像内容

提供详细的图像描述和对象位置信息

pipeline_tag: 图像转文本 tags:

图像描述生成 languages:
英文 license: mit

Kosmos-2：将多模态大语言模型与现实世界锚定

本Hub仓库包含微软原始Kosmos-2模型的HuggingFace transformers实现。

模型使用入门

使用以下代码开始使用模型：

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq


model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

prompt = "<grounding>一张"

url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.png"
image = Image.open(requests.get(url, stream=True).raw)

# 原始Kosmos-2演示会先保存图像再重新加载。对于某些图像，这会略微改变输入并影响生成结果。
image.save("new_image.jpg")
image = Image.open("new_image.jpg")

inputs = processor(text=prompt, images=image, return_tensors="pt")

generated_ids = model.generate(
    pixel_values=inputs["pixel_values"],
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    image_embeds=None,
    image_embeds_position_mask=inputs["image_embeds_position_mask"],
    use_cache=True,
    max_new_tokens=128,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

# 设置`cleanup_and_extract=False`可查看原始模型生成内容。
processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)

print(processed_text)
# `<grounding> 一张<phrase>雪人</phrase><object><patch_index_0044><patch_index_0863></object>在<phrase>火堆</phrase><object><patch_index_0005><patch_index_0911></object>旁取暖的图片`

# 默认会清理生成文本并提取实体。
processed_text, entities = processor.post_process_generation(generated_text)

print(processed_text)
# `一张雪人在火堆旁取暖的图片`

print(entities)
# `[('雪人', (3, 5), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('火堆', (8, 10), [(0.171875, 0.015625, 0.484375, 0.890625)])]`

任务能力

通过修改提示词，该模型可执行不同任务。

首先定义运行提示的函数：

点击展开

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq


model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.png"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(prompt):
    inputs = processor(text=prompt, images=image, return_tensors="pt")
    generated_ids = model.generate(
      pixel_values=inputs["pixel_values"],
      input_ids=inputs["input_ids"],
      attention_mask=inputs["attention_mask"],
      image_embeds=None,
      image_embeds_position_mask=inputs["image_embeds_position_mask"],
      use_cache=True,
      max_new_tokens=128,
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
    _processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)
    processed_text, entities = processor.post_process_generation(generated_text)

    print(processed_text)
    print(entities)
    print(_processed_text)

以下是Kosmos-2能执行的任务示例：

点击展开

多模态锚定

• 短语定位

prompt = "<grounding><phrase>雪人</phrase>"
run_example(prompt)

# 雪人正在火堆旁取暖
# [('雪人', (0, 2), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('火堆', (7, 9), [(0.203125, 0.015625, 0.453125, 0.859375)]]

# <grounding><phrase>雪人</phrase><object><patch_index_0044><patch_index_0863></object>正在<phrase>火堆</phrase><object><patch_index_0006><patch_index_0878></object>旁取暖

• 指代表达理解

prompt = "<grounding><phrase>火堆旁的雪人</phrase>"
run_example(prompt)

# 火堆旁的雪人
# [('火堆旁的雪人', (0, 6), [(0.390625, 0.046875, 0.984375, 0.828125)])]

# <grounding><phrase>火堆旁的雪人</phrase><object><patch_index_0044><patch_index_0863></object>

多模态指代

• 指代表达生成

prompt = "<grounding><phrase>它</phrase><object><patch_index_0044><patch_index_0863></object>是"
run_example(prompt)

# 它是戴帽子和围巾的雪人
# [('它', (0, 1), [(0.390625, 0.046875, 0.984375, 0.828125)])]

# <grounding><phrase>它</phrase><object><patch_index_0044><patch_index_0863></object>是戴帽子和围巾的雪人

感知-语言任务

• 锚定视觉问答

prompt = "<grounding>问题：这张图片有什么特别之处？回答："
run_example(prompt)

# 问题：这张图片有什么特别之处？回答：图中展示了雪地里雪人坐在篝火旁的场景。
# [('雪人', (24, 26), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('篝火', (31, 33), [(0.109375, 0.640625, 0.546875, 0.984375)])]

# <grounding>问题：这张图片有什么特别之处？回答：图中展示了雪地里<phrase>雪人</phrase><object><patch_index_0044><patch_index_0863></object>坐在<phrase>篝火</phrase><object><patch_index_0643><patch_index_1009></object>旁的场景。

• 通过边界框进行多模态指代的视觉问答

prompt = "<grounding>问题：<phrase>火堆</phrase><object><patch_index_0005><patch_index_0911></object>旁边是什么？回答："
run_example(prompt)

# 问题：火堆旁边是什么？回答：雪人附近。
# [('火堆', (3, 5), [(0.171875, 0.015625, 0.484375, 0.890625)]), ('雪人', (12, 14), [(0.390625, 0.046875, 0.984375, 0.828125)])]

# <grounding>问题：<phrase>火堆</phrase><object><patch_index_0005><patch_index_0911></object>旁边是什么？回答：<phrase>雪人</phrase><object><patch_index_0044><patch_index_0863></object>附近。

锚定图像描述

• 简洁描述

prompt = "<grounding>一张"
run_example(prompt)

# 一张雪人在篝火旁取暖的图片
# [('雪人', (2, 4), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('篝火', (7, 9), [(0.109375, 0.640625, 0.546875, 0.984375)])]

# <grounding>一张<phrase>雪人</phrase><object><patch_index_0044><patch_index_0863></object>在<phrase>篝火</phrase><object><patch_index_0643><patch_index_1009></object>旁取暖的图片

• 详细描述

prompt = "<grounding>详细描述这张图片："
run_example(prompt)

# 详细描述这张图片：图中展示了雪地里雪人坐在篝火旁的场景。雪人戴着帽子、围巾和手套，附近有锅和杯子。雪人似乎在享受篝火的温暖，整体氛围温馨舒适。
# [('篝火', (24, 26), [(0.171875, 0.015625, 0.484375, 0.984375)]), ('帽子', (34, 36), [(0.515625, 0.046875, 0.828125, 0.234375)]), ('围巾', (38, 40), [(0.515625, 0.234375, 0.890625, 0.578125)]), ('手套', (43, 45), [(0.515625, 0.390625, 0.640625, 0.515625)]), ('锅', (52, 53), [(0.078125, 0.609375, 0.265625, 0.859375)]), ('杯子', (58, 60), [(0.890625, 0.765625, 0.984375, 0.984375)])]

# <grounding>详细描述这张图片：图中展示了雪地里雪人坐在<phrase>篝火</phrase><object><patch_index_0005><patch_index_1007></object>旁的场景。雪人戴着<phrase>帽子</phrase><object><patch_index_0048><patch_index_0250></object>、<phrase>围巾</phrase><object><patch_index_0240><patch_index_0604></object>和<phrase>手套</phrase><object><patch_index_0400><patch_index_0532></object>，附近有<phrase>锅</phrase><object><patch_index_0610><patch_index_0872></object>和<phrase>杯子</phrase><object><patch_index_0796><patch_index_1023></object>。雪人似乎在享受篝火的温暖，整体氛围温馨舒适。

在图像上绘制实体边界框

获取entities后，可用以下辅助函数在图像上绘制边界框：

点击展开

import cv2
import numpy as np
import os
import requests
import torch
import torchvision.transforms as T

from PIL import Image


def is_overlapping(rect1, rect2):
    x1, y1, x2, y2 = rect1
    x3, y3, x4, y4 = rect2
    return not (x2 < x3 or x1 > x4 or y2 < y3 or y1 > y4)


def draw_entity_boxes_on_image(image, entities, show=False, save_path=None):
    """在图像上绘制实体边界框
    Args:
        image: 图像或图像路径
        collect_entity_location: 实体位置信息
    """
    if isinstance(image, Image.Image):
        image_h = image.height
        image_w = image.width
        image = np.array(image)[:, :, [2, 1, 0]]
    elif isinstance(image, str):
        if os.path.exists(image):
            pil_img = Image.open(image).convert("RGB")
            image = np.array(pil_img)[:, :, [2, 1, 0]]
            image_h = pil_img.height
            image_w = pil_img.width
        else:
            raise ValueError(f"无效图像路径, {image}")
    elif isinstance(image, torch.Tensor):
        image_tensor = image.cpu()
        reverse_norm_mean = torch.tensor([0.48145466, 0.4578275, 0.40821073])[:, None, None]
        reverse_norm_std = torch.tensor([0.26862954, 0.26130258, 0.27577711])[:, None, None]
        image_tensor = image_tensor * reverse_norm_std + reverse_norm_mean
        pil_img = T.ToPILImage()(image_tensor)
        image_h = pil_img.height
        image_w = pil_img.width
        image = np.array(pil_img)[:, :, [2, 1, 0]]
    else:
        raise ValueError(f"无效图像格式, {type(image)} for {image}")

    if len(entities) == 0:
        return image

    new_image = image.copy()
    previous_bboxes = []
    # 文字大小
    text_size = 1
    # 文字线宽
    text_line = 1  # int(max(1 * min(image_h, image_w) / 512, 1))
    box_line = 3
    (c_width, text_height), _ = cv2.getTextSize("F", cv2.FONT_HERSHEY_COMPLEX, text_size, text_line)
    base_height = int(text_height * 0.675)
    text_offset_original = text_height - base_height
    text_spaces = 3

    for entity_name, (start, end), bboxes in entities:
        for (x1_norm, y1_norm, x2_norm, y2_norm) in bboxes:
            orig_x1, orig_y1, orig_x2, orig_y2 = int(x1_norm * image_w), int(y1_norm * image_h), int(x2_norm * image_w), int(y2_norm * image_h)
            # 绘制边界框
            # 随机颜色
            color = tuple(np.random.randint(0, 255, size=3).tolist())
            new_image = cv2.rectangle(new_image, (orig_x1, orig_y1), (orig_x2, orig_y2), color, box_line)

            l_o, r_o = box_line // 2 + box_line % 2, box_line // 2 + box_line % 2 + 1

            x1 = orig_x1 - l_o
            y1 = orig_y1 - l_o

            if y1 < text_height + text_offset_original + 2 * text_spaces:
                y1 = orig_y1 + r_o + text_height + text_offset_original + 2 * text_spaces
                x1 = orig_x1 + r_o

            # 添加文字背景
            (text_width, text_height), _ = cv2.getTextSize(f"  {entity_name}", cv2.FONT_HERSHEY_COMPLEX, text_size, text_line)
            text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2 = x1, y1 - (text_height + text_offset_original + 2 * text_spaces), x1 + text_width, y1

            for prev_bbox in previous_bboxes:
                while is_overlapping((text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2), prev_bbox):
                    text_bg_y1 += (text_height + text_offset_original + 2 * text_spaces)
                    text_bg_y2 += (text_height + text_offset_original + 2 * text_spaces)
                    y1 += (text_height + text_offset_original + 2 * text_spaces)

                    if text_bg_y2 >= image_h:
                        text_bg_y1 = max(0, image_h - (text_height + text_offset_original + 2 * text_spaces))
                        text_bg_y2 = image_h
                        y1 = image_h
                        break

            alpha = 0.5
            for i in range(text_bg_y1, text_bg_y2):
                for j in range(text_bg_x1, text_bg_x2):
                    if i < image_h and j < image_w:
                        if j < text_bg_x1 + 1.35 * c_width:
                            # 原始颜色
                            bg_color = color
                        else:
                            # 白色
                            bg_color = [255, 255, 255]
                        new_image[i, j] = (alpha * new_image[i, j] + (1 - alpha) * np.array(bg_color)).astype(np.uint8)

            cv2.putText(
                new_image, f"  {entity_name}", (x1, y1 - text_offset_original - 1 * text_spaces), cv2.FONT_HERSHEY_COMPLEX, text_size, (0, 0, 0), text_line, cv2.LINE_AA
            )
            # previous_locations.append((x1, y1))
            previous_bboxes.append((text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2))

    pil_image = Image.fromarray(new_image[:, :, [2, 1, 0]])
    if save_path:
        pil_image.save(save_path)
    if show:
        pil_image.show()

    return new_image


# (使用之前代码示例中的相同图像)
url = "