许可协议：apache-2.0
任务类型：掩码生成
库名称：sam2

SAM 2（Segment Anything in Images and Videos）代码库，这是FAIR研发的面向图像与视频可提示视觉分割的基础模型。更多信息请参阅SAM 2论文。

官方代码已发布于该代码库。

使用说明

图像预测：

import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<您的图像>)
    masks, _, _ = predictor.predict(<输入提示>)

视频预测：

import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<您的视频>)

    # 添加新提示并立即获取当前帧输出
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <您的提示>):

    # 传播提示以获取视频全程的掩码片段
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

详情请参考演示笔记本。

引用

如需引用论文、模型或软件，请使用以下格式：

@article{ravi2024sam2,
  title={SAM 2：图像与视频的通用分割},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv预印本 arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}