许可证：apache-2.0
语言：

• 英文
  任务标签：视觉问答
  标签：
• 聊天

mPLUG-Owl3

简介

mPLUG-Owl3 是一款先进的多模态大语言模型，旨在解决长图像序列理解的挑战。我们提出了超注意力机制（Hyper Attention），将多模态大语言模型中长视觉序列的理解速度提升六倍，并能处理长度达八倍的视觉序列。同时，该模型在单图、多图和视频任务中均保持卓越性能。

GitHub：mPLUG-Owl

快速开始

加载 mPLUG-Owl3。目前仅支持 attn_implementation 为 ['sdpa', 'flash_attention_2'] 的配置。

import torch  
from transformers import AutoConfig, AutoModel  
model_path = 'mPLUG/mPLUG-Owl3-1B-241014'  
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)  
print(config)  
# model = mPLUGOwl3Model(config).cuda().half()  
model = AutoModel.from_pretrained(model_path, attn_implementation='sdpa', torch_dtype=torch.half, trust_remote_code=True)  
model.eval().cuda()  

与图像对话。

from PIL import Image  
from transformers import AutoTokenizer, AutoProcessor  
from decord import VideoReader, cpu  
model_path = 'mPLUG/mPLUG-Owl3-1B-241014'  
tokenizer = AutoTokenizer.from_pretrained(model_path)  
processor = model.init_processor(tokenizer)  

image = Image.new('RGB', (500, 500), color='red')  

messages = [  
    {"role": "user", "content": """<|image|>  
描述这张图片。"""},  
    {"role": "assistant", "content": ""}  
]  

inputs = processor(messages, images=[image], videos=None)  
inputs.to('cuda')  
inputs.update({  
    'tokenizer': tokenizer,  
    'max_new_tokens': 100,  
    'decode_text': True,  
})  

g = model.generate(**inputs)  
print(g)  

与视频对话。

from PIL import Image  
from transformers import AutoTokenizer, AutoProcessor  
from decord import VideoReader, cpu    # pip install decord  
model_path = 'mPLUG/mPLUG-Owl3-1B-241014'  
tokenizer = AutoTokenizer.from_pretrained(model_path)  
processor = model.init_processor(tokenizer)  

messages = [  
    {"role": "user", "content": """<|video|>  
描述这段视频。"""},  
    {"role": "assistant", "content": ""}  
]  

videos = ['/nas-mmu-data/examples/car_room.mp4']  
MAX_NUM_FRAMES = 16  

def encode_video(video_path):  
    def uniform_sample(l, n):  
        gap = len(l) / n  
        idxs = [int(i * gap + gap / 2) for i in range(n)]  
        return [l[i] for i in idxs]  

    vr = VideoReader(video_path, ctx=cpu(0))  
    sample_fps = round(vr.get_avg_fps() / 1)  # 帧率  
    frame_idx = [i for i in range(0, len(vr), sample_fps)]  
    if len(frame_idx) > MAX_NUM_FRAMES:  
        frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)  
    frames = vr.get_batch(frame_idx).asnumpy()  
    frames = [Image.fromarray(v.astype('uint8')) for v in frames]  
    print('帧数:', len(frames))  
    return frames  

video_frames = [encode_video(_) for _ in videos]  
inputs = processor(messages, images=None, videos=video_frames)  
inputs.to('cuda')  
inputs.update({  
    'tokenizer': tokenizer,  
    'max_new_tokens': 100,  
    'decode_text': True,  
})  

g = model.generate(**inputs)  
print(g)  

引用

如果您觉得我们的工作有帮助，欢迎引用。

@misc{ye2024mplugowl3longimagesequenceunderstanding,  
      title={mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models},   
      author={Jiabo Ye and Haiyang Xu and Haowei Liu and Anwen Hu and Ming Yan and Qi Qian and Ji Zhang and Fei Huang and Jingren Zhou},  
      year={2024},  
      eprint={2408.04840},  
      archivePrefix={arXiv},  
      primaryClass={cs.CV},  
      url={https://arxiv.org/abs/2408.04840},   
}