许可证：apache-2.0
任务标签：图像转文本
数据集：

• hoang-quoc-trung/fusion-image-to-latex-datasets
  标签：
• img2latex
• latex ocr
• 印刷数学公式识别
• 手写数学公式识别

数学公式图像转LaTeX序列

提升图像转LaTeX性能：基于大规模数据集的端到端Transformer模型Sumen

性能表现

使用方式

源代码：https://github.com/hoang-quoc-trung/sumen

推理示例

import torch  
import requests  
from PIL import Image  
from transformers import AutoProcessor, VisionEncoderDecoderModel  

# 加载模型与处理器  
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  
model = VisionEncoderDecoderModel.from_pretrained('hoang-quoc-trung/sumen-base').to(device)  
processor = AutoProcessor.from_pretrained('hoang-quoc-trung/sumen-base')  
task_prompt = processor.tokenizer.bos_token  
decoder_input_ids = processor.tokenizer(  
    task_prompt,  
    add_special_tokens=False,  
    return_tensors="pt"  
).input_ids  

# 加载图像  
img_url = 'https://raw.githubusercontent.com/hoang-quoc-trung/sumen/main/assets/example_1.png'  
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')  
pixel_values = processor.image_processor(  
    image,  
    return_tensors="pt",  
    data_format="channels_first",  
).pixel_values  

# 生成LaTeX表达式  
with torch.no_grad():  
    outputs = model.generate(  
        pixel_values.to(device),  
        decoder_input_ids=decoder_input_ids.to(device),  
        max_length=model.decoder.config.max_length,  
        pad_token_id=processor.tokenizer.pad_token_id,  
        eos_token_id=processor.tokenizer.eos_token_id,  
        use_cache=True,  
        num_beams=4,  
        bad_words_ids=[[processor.tokenizer.unk_token_id]],  
        return_dict_in_generate=True,  
    )  

sequence = processor.tokenizer.batch_decode(outputs.sequences)[0]  
sequence = sequence.replace(  
        processor.tokenizer.eos_token, ""  
    ).replace(  
        processor.tokenizer.pad_token, ""  
    ).replace(processor.tokenizer.bos_token,"")  
print(sequence)