license: apache-2.0
language: en

图表转表格模型

图表转表格模型（Chart-To-Table）首次发表于论文《Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning》，用于将图表转换为结构化表格。生成的表格使用&&&分隔行，|分隔列。该模型基于UniChart架构构建。

使用方法

from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image

model_name = "khhuang/chart-to-table"
model = VisionEncoderDecoderModel.from_pretrained(model_name).cuda()
processor = DonutProcessor.from_pretrained(model_name)

image_path = "图片路径"

# 格式化文本输入
input_prompt = "<data_table_generation> <s_answer>"

# 编码图表图像并分词文本
img = Image.open(图片路径)
pixel_values = processor(img.convert("RGB"), random_padding=False, return_tensors="pt").pixel_values
pixel_values = pixel_values.cuda()
decoder_input_ids = processor.tokenizer(input_prompt, add_special_tokens=False, return_tensors="pt", max_length=510).input_ids.cuda()


outputs = model.generate(
        pixel_values.cuda(),
        decoder_input_ids=decoder_input_ids.cuda(),
        max_length=model.decoder.config.max_position_embeddings,
        early_stopping=True,
        pad_token_id=processor.tokenizer.pad_token_id,
        eos_token_id=processor.tokenizer.eos_token_id,
        use_cache=True,
        num_beams=4,
        bad_words_ids=[[processor.tokenizer.unk_token_id]],
        return_dict_in_generate=True,
    )
    

sequence = processor.batch_decode(outputs.sequences)[0]
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
extracted_table = sequence.split("<s_answer>")[1].strip()

引用

@misc{huang-etal-2023-do,
    title = "多模态大语言模型理解图表吗？图表描述中事实性错误的分析与修正",
    author = "黄孔祥 和
      周明阳 和
      陈厚鹏 和
      冯艺茹 和
      王振海龙 和
      张凌宇 和
      张世富 和
      纪珩",
    year={2023},
    eprint={2312.10160},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}