许可协议: apache-2.0
标签:

• 视觉
• dinov2
• 深度估计
  推理: false

模型卡片: 基于DINOv2骨干网络的DPT模型

模型详情

DPT（密集预测变换器）模型采用DINOv2骨干网络，如Oquab等人在论文DINOv2: 无监督学习鲁棒视觉特征中所提出的。

DPT架构图。摘自原论文。

资源

• DINOv2论文
• DPT论文

与Transformers结合使用

from transformers import AutoImageProcessor, DPTForDepthEstimation  
import torch  
import numpy as np  
from PIL import Image  
import requests  

url = "http://images.cocodataset.org/val2017/000000039769.jpg"  
image = Image.open(requests.get(url, stream=True).raw)  

image_processor = AutoImageProcessor.from_pretrained("facebook/dpt-dinov2-large-nyu")  
model = DPTForDepthEstimation.from_pretrained("facebook/dpt-dinov2-large-nyu")  

# 为模型准备图像  
inputs = image_processor(images=image, return_tensors="pt")  

with torch.no_grad():  
    outputs = model(**inputs)  
    predicted_depth = outputs.predicted_depth  

# 插值到原始尺寸  
prediction = torch.nn.functional.interpolate(  
    predicted_depth.unsqueeze(1),  
    size=image.size[::-1],  
    mode="bicubic",  
    align_corners=False,  
)  

# 可视化预测结果  
output = prediction.squeeze().cpu().numpy()  
formatted = (output * 255 / np.max(output)).astype("uint8")  
depth = Image.fromarray(formatted)  

模型用途

预期用途

该模型旨在展示使用DPT框架结合DINOv2骨干网络可以构建一个强大的深度估计器。

BibTeX条目及引用信息

@misc{oquab2023dinov2,  
      title={DINOv2: Learning Robust Visual Features without Supervision},   
      author={Maxime Oquab and Timothée Darcet and Théo Moutakanni and Huy Vo and Marc Szafraniec and Vasil Khalidov and Pierre Fernandez and Daniel Haziza and Francisco Massa and Alaaeldin El-Nouby and Mahmoud Assran and Nicolas Ballas and Wojciech Galuba and Russell Howes and Po-Yao Huang and Shang-Wen Li and Ishan Misra and Michael Rabbat and Vasu Sharma and Gabriel Synnaeve and Hu Xu and Hervé Jegou and Julien Mairal and Patrick Labatut and Armand Joulin and Piotr Bojanowski},  
      year={2023},  
      eprint={2304.07193},  
      archivePrefix={arXiv},  
      primaryClass={cs.CV}  
}