许可证：apache-2.0
库名称：timm
标签：

• 图像特征提取
• timm
• transformers

SAM ViT大尺寸分块16.sa1b模型卡

一款Segment-Anything视觉Transformer（SAM ViT）图像特征模型（注：仅包含特征提取和微调功能，未包含分割头）。由论文作者基于SA-1B数据集进行预训练用于分割任务，权重初始化采用MAE预训练权重。

模型详情

• 模型类型： 图像分类/特征主干网络
• 模型统计：
  • 参数量（百万）：308.3
  • 计算量（GMACs）：1493.9
  • 激活值（百万）：2553.8
  • 图像尺寸：1024 x 1024
• 相关论文：
  • 《Segment Anything》：https://arxiv.org/abs/2304.02643
  • 《An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale》：https://arxiv.org/abs/2010.11929v2
• 原始代码库： https://github.com/facebookresearch/segment-anything
• 预训练数据集： SA-1B

模型使用

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('samvit_large_patch16.sa1b', pretrained=True)
model = model.eval()

# 获取模型特定的数据变换（标准化、尺寸调整）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 将单张图像扩展为批大小为1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'samvit_large_patch16.sa1b',
    pretrained=True,
    num_classes=0,  # 移除分类器nn.Linear
)
model = model.eval()

# 获取模型特定的数据变换（标准化、尺寸调整）
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # 输出为(batch_size, num_features)形状张量

# 或等效操作（无需设置num_classes=0）

output = model.forward_features(transforms(img).unsqueeze(0))
# 输出为非池化结果，形状为(1, 256, 64, 64)的张量

output = model.forward_head(output, pre_logits=True)
# 输出为(1, num_features)形状张量

模型对比

在timm的模型结果中探索该模型的数据集和运行时指标。

引用文献

@article{kirillov2023segany,
  title={Segment Anything},
  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{'a}r, Piotr and Girshick, Ross},
  journal={arXiv:2304.02643},
  year={2023}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}