license: apache-2.0 library_name: timm tags:

• 图像特征提取
• timm
• transformers

vit_small_patch8_224.dino 模型卡片

一个基于视觉Transformer（ViT）的图像特征模型。采用自监督DINO方法训练。

模型详情

• 模型类型: 图像分类/特征骨干网络
• 模型统计:
  • 参数量（百万）: 21.7
  • GMACs运算量: 16.8
  • 激活值（百万）: 32.9
  • 图像尺寸: 224 x 224
• 相关论文:
  • 《自监督视觉Transformer的新兴特性》: https://arxiv.org/abs/2104.14294
  • 《一张图片等于16x16个单词：大规模图像识别的Transformer应用》: https://arxiv.org/abs/2010.11929v2
• 预训练数据集: ImageNet-1k
• 原始实现: https://github.com/facebookresearch/dino

模型使用

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_small_patch8_224.dino', pretrained=True)
model = model.eval()

# 获取模型特定的变换（归一化、调整尺寸）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 将单张图片扩展为批大小为1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_small_patch8_224.dino',
    pretrained=True,
    num_classes=0,  # 移除分类器nn.Linear
)
model = model.eval()

# 获取模型特定的变换（归一化、调整尺寸）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 输出为（批大小，特征数量）形状的张量

# 或等效操作（无需设置num_classes=0）

output = model.forward_features(transforms(img).unsqueeze(0))
# 输出为非池化结果，形状为（1，785，384）的张量

output = model.forward_head(output, pre_logits=True)
# 输出为（1，特征数量）形状的张量

模型比较

在timm的模型结果中探索该模型的数据集和运行时指标。

引用

@inproceedings{caron2021emerging,
  title={自监督视觉Transformer的新兴特性},
  author={Caron, Mathilde and Touvron, Hugo and Misra, Ishan and J{'e}gou, Herv{'e} and Mairal, Julien and Bojanowski, Piotr and Joulin, Armand},
  booktitle={IEEE/CVF国际计算机视觉会议论文集},
  pages={9650--9660},
  year={2021}
}

@article{dosovitskiy2020vit,
  title={一张图片等于16x16个单词：大规模图像识别的Transformer应用},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch图像模型库},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub仓库},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}