许可证：Apache-2.0
语言：

• 英文
  任务标签：图像文本到文本
  标签：
• 聊天

mPLUG-DocOwl2

简介

mPLUG-DocOwl2 是一款先进的无需OCR的多页文档理解多模态大语言模型。

通过名为“高分辨率文档压缩器”的压缩模块，每页文档仅用324个标记进行编码。

GitHub：mPLUG-DocOwl

快速开始

import torch  
import os  
from transformers import AutoTokenizer, AutoModel  
from icecream import ic  
import time  

class DocOwlInfer():  
    def __init__(self, ckpt_path):  
        self.tokenizer = AutoTokenizer.from_pretrained(ckpt_path, use_fast=False)  
        self.model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True, low_cpu_mem_usage=True, torch_dtype=torch.float16, device_map='auto')  
        self.model.init_processor(tokenizer=self.tokenizer, basic_image_size=504, crop_anchors='grid_12')  

    def inference(self, images, query):  
        messages = [{'role': 'USER', 'content': '<|image|>'*len(images)+query}]  
        answer = self.model.chat(messages=messages, images=images, tokenizer=self.tokenizer)  
        return answer  

docowl = DocOwlInfer(ckpt_path='mPLUG/DocOwl2')  

images = [  
        './examples/docowl2_page0.png',  
        './examples/docowl2_page1.png',  
        './examples/docowl2_page2.png',  
        './examples/docowl2_page3.png',  
        './examples/docowl2_page4.png',  
        './examples/docowl2_page5.png',  
    ]  

answer = docowl.inference(images, query='这篇论文是关于什么的？请提供详细信息。')  

answer = docowl.inference(images, query='第三页是关于什么的？请提供详细信息。')