许可证: mit 数据集:

• laion/laion2B-en
• laion/laion-coco
• laion/laion2B-multi
• kakaobrain/coyo-700m
• conceptual_captions
• wanng/wukong100m 管道标签: 图像特征提取 基础模型: OpenGVLab/InternViT-6B-448px-V1-0 基础模型关系: 微调 新版本: OpenGVLab/InternViT-6B-448px-V2_5

InternViT-6B-448px-V1-2

[📂 GitHub] [📜 InternVL 1.0] [📜 InternVL 1.5] [📜 Mini-InternVL] [📜 InternVL 2.5]

[🆕 博客] [🗨️ 聊天演示] [🤗 HF 演示] [🚀 快速开始] [📖 文档]

我们发布了新的InternViT权重InternViT-6B-448px-V1-2。InternViT-6B模型的持续预训练参与了InternVL 1.2的更新。具体来说，我们将InternViT-6B的分辨率从224提升到448，并将其与[Nous-Hermes-2-Yi-34B]((https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)集成。为了赋予模型高分辨率处理和OCR能力，视觉编码器和MLP都被激活进行训练，利用了图像描述和OCR特定数据集的混合。

模型详情

• 模型类型: 视觉基础模型，特征骨干
• 模型统计:
  • 参数（百万）: 5540（最后3个块被丢弃）
  • 图像尺寸: 448 x 448
• 预训练数据集: LAION-en, LAION-zh, COYO, GRIT, COCO, TextCaps, Objects365, OpenImages, All-Seeing, Wukong-OCR, LaionCOCO-OCR以及其他OCR相关数据集。为了增强模型的OCR能力，我们在通用描述数据集之外还加入了额外的OCR数据。具体来说，我们使用PaddleOCR对Wukong中的图像进行中文OCR，对LAION-COCO中的图像进行英文OCR。
• 注意: InternViT-6B原本有48个块，我们发现使用倒数第四个块的输出对于MLLM效果最佳。为了方便使用并节省GPU内存，我们简单地丢弃了最后3个块。现在，模型只有45个块，参数数量从5.9B减少到5.5B。因此，如果你想基于此模型构建MLLM，请使用最后一层的特征。

快速开始

[!警告] 🚨 注意：根据我们的经验，InternViT V2.5系列更适合构建MLLM，而不是传统的计算机视觉任务。

import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

model = AutoModel.from_pretrained(
    'OpenGVLab/InternViT-6B-448px-V1-2',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).cuda().eval()

image = Image.open('./examples/image1.jpg').convert('RGB')

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-2')

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()

outputs = model(pixel_values)

许可证

本项目采用MIT许可证发布。

引用

如果你在研究中发现本项目有用，请考虑引用：

@article{chen2024expanding,
  title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},
  author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},
  journal={arXiv preprint arXiv:2412.05271},
  year={2024}
}
@article{gao2024mini,
  title={Mini-internvl: A flexible-transfer pocket multimodal model with 5\% parameters and 90\% performance},
  author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},
  journal={arXiv preprint arXiv:2410.16261},
  year={2024}
}
@article{chen2024far,
  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
  journal={arXiv preprint arXiv:2404.16821},
  year={2024}
}
@inproceedings{chen2024internvl,
  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={24185--24198},
  year={2024}
}