库名称: transformers 许可证: apple-amlr 流水线标签: 零样本图像分类 标签:

• 视觉
• mlx
• pytorch

简介

[AIMv2论文] [BibTeX]

我们推出了采用多模态自回归目标预训练的AIMv2系列视觉模型。AIMv2预训练方法简单直接，易于训练且能有效扩展。AIMv2的主要亮点包括：

1. 在多数多模态理解基准测试中超越OAI CLIP和SigLIP
2. 在开放词汇目标检测和指代表达理解任务上优于DINOv2
3. 展现强大识别性能——AIMv2-3B在使用冻结主干网络时，在ImageNet上达到89.5%准确率

使用指南

PyTorch

import requests
from PIL import Image
from transformers import AutoProcessor, AutoModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = ["狗的图片", "猫的图片", "马的图片"]

processor = AutoProcessor.from_pretrained(
    "apple/aimv2-large-patch14-224-lit",
)
model = AutoModel.from_pretrained(
    "apple/aimv2-large-patch14-224-lit",
    trust_remote_code=True,
)

inputs = processor(
    images=image,
    text=text,
    add_special_tokens=True,
    truncation=True,
    padding=True,
    return_tensors="pt",
)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=-1)

JAX

开发中。

引用

如果您认为我们的工作有价值，请考虑引用：

@misc{fini2024multimodalautoregressivepretraininglarge,
  author      = {Fini, Enrico and Shukor, Mustafa and Li, Xiujun and Dufter, Philipp and Klein, Michal and Haldimann, David and Aitharaju, Sai and da Costa, Victor Guilherme Turrisi and Béthune, Louis and Gan, Zhe and Toshev, Alexander T and Eichner, Marcin and Nabi, Moin and Yang, Yinfei and Susskind, Joshua M. and El-Nouby, Alaaeldin},
  url         = {https://arxiv.org/abs/2411.14402},
  eprint      = {2411.14402},
  eprintclass = {cs.CV},
  eprinttype  = {arXiv},
  title       = {Multimodal Autoregressive Pre-training of Large Vision Encoders},
  year        = {2024},
}