许可协议: mit
数据集:

• mlfoundations/datacomp_1b
  管道标签: 特征提取

ViTamin-XL-256px模型卡片

ViTamin的官方HuggingFace模型，源自以下CVPR 2024论文：

ViTamin：视觉语言时代下的可扩展视觉模型设计。
✨  陈杰能、俞启航、沈晓晖、艾伦·尤尔和陈亮杰
🏠  约翰霍普金斯大学，字节跳动

使用transformers.AutoModel从HuggingFace加载：

import torch  
import open_clip  
from PIL import Image  
from transformers import AutoModel, CLIPImageProcessor  
device = "cuda" if torch.cuda.is_available() else "cpu"  

model = AutoModel.from_pretrained(  
    'jienengchen/ViTamin-XL-256px',  
    trust_remote_code=True).to(device).eval()  

image = Image.open('./image.png').convert('RGB')  
image_processor = CLIPImageProcessor.from_pretrained('jienengchen/ViTamin-XL-256px')  

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values  
pixel_values = pixel_values.to(torch.bfloat16).cuda()  

tokenizer = open_clip.get_tokenizer('hf-hub:laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K')  
text = tokenizer(["维生素的照片", "一只狗", "一只猫"]).to(device)  

with torch.no_grad(), torch.cuda.amp.autocast():  
    image_features, text_features, logit_scale = model(pixel_values, text)  
    text_probs = (100.0 * image_features @ text_features.to(torch.float).T).softmax(dim=-1)  

print("标签概率:", text_probs)  

基于DataComp-1B的CLIP预训练主要结果

下游任务主要结果

开放词汇检测

开放词汇分割

注：全景数据集（ADE、CityScapes、MV）使用PQ指标。语义数据集（A-150、A-847、PC-459、PC-59、PAS-21）使用mIoU指标。

大型多模态模型

引用ViTamin

@inproceedings{chen2024vitamin,  
  title={ViTamin: Design Scalable Vision Models in the Vision-language Era},  
  author={Chen, Jieneng and Yu, Qihang and Shen, Xiaohui and Yuille, ALan and Chen, Liang-Chieh},  
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},  
  year={2024}  
}