数据集:

• mlfoundations/datacomp_1b 许可证: mit 标签:
• pytorch_model_hub_mixin
• model_hub_mixin 任务类型: zero-shot-image-classification 库名称: prolip

基于DataComp 1B预训练的ViT-B/16 ProLIP官方实现

• 本权重为概率语言-图像预训练(ProLIP)方法预训练的ViT-B/16模型
• 预训练数据集
  • DataComp 1B / 实际训练样本12.8B

概览

• 论文: https://arxiv.org/abs/2410.18857
• GitHub: https://github.com/naver-ai/prolip
• 更多模型请访问: https://huggingface.co/collections/SanghyukChun/prolip-6712595dfc87fd8597350291

性能概览

• 零样本ImageNet-1k top-1准确率: 74.6%
• 零样本ImageNet分布偏移: 63.0%
• 零样本VTAB性能: 63.7%
• 零样本检索性能: 59.6%
• 38项任务平均零样本性能: 63.3%

import requests
from PIL import Image

import torch
from prolip.model import ProLIPHF
from transformers import CLIPProcessor
from prolip.tokenizer import HFTokenizer

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch16")
model = ProLIPHF.from_pretrained("SanghyukChun/ProLIP-ViT-B-16-DC-1B-12_8M")
tokenizer = HFTokenizer("timm/ViT-B-16-SigLIP", context_length=64, clean="canonicalize")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt", padding=True)
texts = ["两只猫躺在粉色毯子上", "暴雨中男子涉水过马路", "照片"]
texts = tokenizer(texts)

outputs = model(image=inputs["pixel_values"], text=texts)

l2_logit = outputs["image_features"]["mean"] @ outputs["text_features"]["mean"].T
i_unc = torch.exp(outputs["image_features"]["std"]).sum(dim=-1)
t_unc = torch.exp(outputs["text_features"]["std"]).sum(dim=-1)
csd_logit = l2_logit - 0.5 * t_unc
csd_logit2 = l2_logit.T - 0.5 * i_unc
print("仅均值图像-文本对数概率(L2距离):", l2_logit)
print("不确定性感知图像-文本对数概率(CSD):", csd_logit)
print("不确定性感知文本-图像对数概率(CSD):", csd_logit2.T)
print("图像不确定性: ", i_unc)
print("文本不确定性: ", t_unc)

引用文献

@inproceedings{chun2025prolip,
    title={概率语言-图像预训练},
    author={Chun, Sanghyuk and Kim, Wonjae and Park, Song and Yun, Sangdoo},
    year={2025},
    booktitle={国际学习表征会议(ICLR)},
}

@inproceedings{chun2025longprolip,
    title={LongProLIP: 支持长文本的概率视觉语言模型},
    author={Chun, Sanghyuk and Yun, Sangdoo},
    year={2025},
    booktitle={ICLR基础模型不确定性与幻觉量化研讨会},
}