标签:

• siglip
• siglip2
• 视觉 库名称: open_clip 管道标签: 零样本图像分类 许可证: apache-2.0 数据集:
• webli

ViT-SO400M-16-SigLIP2-384 模型卡

模型详情

一个在WebLI数据集上训练的SigLIP 2视觉语言模型。

该模型已从Big Vision的原始JAX检查点转换为适用于OpenCLIP的格式。

模型详情

• 模型类型: 对比图像-文本，零样本图像分类。
• 原始来源: https://github.com/google-research/big_vision
• 数据集: WebLI
• 相关论文:
  • SigLIP 2：具有改进语义理解、定位和密集特征的多语言视觉语言编码器: https://arxiv.org/abs/2502.14786
  • 用于语言图像预训练的Sigmoid损失函数: https://arxiv.org/abs/2303.15343

模型使用

import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer # 需要open-clip-torch >= 2.31.0, timm >= 1.0.15

model, preprocess = create_model_from_pretrained('hf-hub:timm/ViT-SO400M-16-SigLIP2-384')
tokenizer = get_tokenizer('hf-hub:timm/ViT-SO400M-16-SigLIP2-384')

image = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))
image = preprocess(image).unsqueeze(0)

labels_list = ["一只狗", "一只猫", "一个甜甜圈", "一个贝涅饼"]
text = tokenizer(labels_list, context_length=model.context_length)

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    text_probs = torch.sigmoid(image_features @ text_features.T * model.logit_scale.exp() + model.logit_bias)

zipped_list = list(zip(labels_list, [100 * round(p.item(), 3) for p in text_probs[0]]))
print("标签概率: ", zipped_list)

引用

@article{tschannen2025siglip,
  title={SigLIP 2：具有改进语义理解、定位和密集特征的多语言视觉语言编码器},
  author={Tschannen, Michael and Gritsenko, Alexey and Wang, Xiao and Naeem, Muhammad Ferjad and Alabdulmohsin, Ibrahim and Parthasarathy, Nikhil and Evans, Talfan and Beyer, Lucas and Xia, Ye and Mustafa, Basil and H'enaff, Olivier and Harmsen, Jeremiah and Steiner, Andreas and Zhai, Xiaohua},
  year={2025},
  journal={arXiv预印本 arXiv:2502.14786}
}        

@article{zhai2023sigmoid,
  title={用于语言图像预训练的Sigmoid损失函数},
  author={Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  journal={arXiv预印本 arXiv:2303.15343},
  year={2023}
}

@misc{big_vision,
  author = {Beyer, Lucas and Zhai, Xiaohua and Kolesnikov, Alexander},
  title = {Big Vision},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub仓库},
  howpublished = {\url{https://github.com/google-research/big_vision}}
}