license: apache-2.0 library_name: timm tags:

• 图像特征提取
• timm
• transformers

vit_base_patch14_reg4_dinov2.lvd142m 模型卡

一个带有寄存器的视觉变换器（ViT）图像特征模型。使用自监督的DINOv2方法在LVD-142M数据集上进行预训练。

模型详情

• 模型类型: 图像分类 / 特征骨干网络
• 模型统计:
  • 参数（百万）: 86.6
  • GMACs: 117.5
  • 激活（百万）: 115.0
  • 图像尺寸: 518 x 518
• 相关论文:
  • Vision Transformers Need Registers: https://arxiv.org/abs/2309.16588
  • DINOv2: Learning Robust Visual Features without Supervision: https://arxiv.org/abs/2304.07193
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929v2
• 原始代码: https://github.com/facebookresearch/dinov2
• 预训练数据集: LVD-142M

模型使用

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_base_patch14_reg4_dinov2.lvd142m', pretrained=True)
model = model.eval()

# 获取模型特定的变换（归一化、调整大小）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 将单张图像扩展为批次大小为1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_base_patch14_reg4_dinov2.lvd142m',
    pretrained=True,
    num_classes=0,  # 移除分类器 nn.Linear
)
model = model.eval()

# 获取模型特定的变换（归一化、调整大小）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 输出是形状为（batch_size, num_features）的张量

# 或者等效地（无需设置 num_classes=0）

output = model.forward_features(transforms(img).unsqueeze(0))
# 输出是非池化的，形状为（1, 1374, 768）的张量

output = model.forward_head(output, pre_logits=True)
# 输出是形状为（1, num_features）的张量

模型比较

在timm的模型结果中探索该模型的数据集和运行时指标。

引用

@article{darcet2023vision,
  title={Vision Transformers Need Registers},
  author={Darcet, Timoth{'e}e and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr},
  journal={arXiv preprint arXiv:2309.16588},
  year={2023}
}

@misc{oquab2023dinov2,
  title={DINOv2: Learning Robust Visual Features without Supervision},
  author={Oquab, Maxime and Darcet, Timothée and Moutakanni, Theo and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Howes, Russell and Huang, Po-Yao and Xu, Hu and Sharma, Vasu and Li, Shang-Wen and Galuba, Wojciech and Rabbat, Mike and Assran, Mido and Ballas, Nicolas and Synnaeve, Gabriel and Misra, Ishan and Jegou, Herve and Mairal, Julien and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr},
  journal={arXiv:2304.07193},
  year={2023}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}