缩略图: "用于社交分享的缩略图链接" 标签:

• 标签1
• 标签2 许可证: apache-2.0 数据集:
• 数据集1
• 数据集2 评估指标:
• 指标1
• 指标2

模型架构:

mychen76/mistral7b_ocr_to_json_v1（LLM）是针对将OCR文本转换为JSON对象任务进行微调的模型。这一实验性模型基于Mistral-7B-v0.1，在所有测试基准上均优于Llama 2 13B。

动机:

目前，OCR引擎在图像检测和文本识别方面已经过充分测试，而LLM模型在文本处理和生成方面训练有素。因此，利用OCR引擎的输出可以节省LLM在图像到文本用例（如发票或收据图像到JSON对象转换任务）上的训练时间。

模型使用:

拍摄发票或收据图像，对图像进行OCR处理以获取文本框，并将输出输入LLM模型以生成格式良好的收据JSON对象。

### 指令:
您是POS收据数据专家，请解析、检测、识别并将以下收据OCR图像结果转换为结构化的收据数据对象。
不要编造输入中没有的值。输出必须是一个格式良好的JSON对象。```json

### 输入:
[[[[184.0, 42.0], [278.0, 45.0], [278.0, 62.0], [183.0, 59.0]], ('BAJA FRESH', 0.9551795721054077)], [[[242.0, 113.0], [379.0, 118.0], [378.0, 136.0], [242.0, 131.0]], ('GENERAL MANAGER:', 0.9462024569511414)], [[[240.0, 133.0], [300.0, 135.0], [300.0, 153.0], [240.0, 151.0]], ('NORMAN', 0.9913229942321777)], [[[143.0, 166.0], [234.0, 171.0], [233.0, 192.0], [142.0, 187.0]], ('176 Rosa C', 0.9229503870010376)], [[[130.0, 207.0], [206.0, 210.0], [205.0, 231.0], [129.0, 228.0]], ('Chk 7545', 0.9349349141120911)], [[[283.0, 215.0], [431.0, 221.0], [431.0, 239.0], [282.0, 233.0]], ("Dec26'0707:26PM", 0.9290117025375366)], [[[440.0, 221.0], [489.0, 221.0], [489.0, 239.0], [440.0, 239.0]], ('Gst0', 0.9164432883262634)], [[[164.0, 252.0], [308.0, 256.0], [308.0, 276.0], [164.0, 272.0]], ('TAKE OUT', 0.9367803335189819)], [[[145.0, 274.0], [256.0, 278.0], [255.0, 296.0], [144.0, 292.0]], ('1 BAJA STEAK', 0.9167789816856384)], [[[423.0, 282.0], [465.0, 282.0], [465.0, 304.0], [423.0, 304.0]], ('6.95', 0.9965073466300964)], [[[180.0, 296.0], [292.0, 299.0], [292.0, 319.0], [179.0, 316.0]], ('NO GUACAMOLE', 0.9631438255310059)], [[[179.0, 317.0], [319.0, 322.0], [318.0, 343.0], [178.0, 338.0]], ('ENCHILADO STYLE', 0.9704310894012451)], [[[423.0, 325.0], [467.0, 325.0], [467.0, 347.0], [423.0, 347.0]], ('1.49', 0.988395631313324)], [[[159.0, 339.0], [201.0, 341.0], [200.0, 360.0], [158.0, 358.0]], ('CASH', 0.9982023239135742)], [[[417.0, 348.0], [466.0, 348.0], [466.0, 367.0], [417.0, 367.0]], ('20.00', 0.9921982884407043)], [[[156.0, 380.0], [200.0, 382.0], [198.0, 404.0], [155.0, 402.0]], ('FOOD', 0.9906187057495117)], [[[426.0, 390.0], [468.0, 390.0], [468.0, 409.0], [426.0, 409.0]], ('8.44', 0.9963030219078064)], [[[154.0, 402.0], [190.0, 405.0], [188.0, 427.0], [152.0, 424.0]], ('TAX', 0.9963871836662292)], [[[427.0, 413.0], [468.0, 413.0], [468.0, 432.0], [427.0, 432.0]], ('0.61', 0.9934712648391724)], [[[153.0, 427.0], [224.0, 429.0], [224.0, 450.0], [153.0, 448.0]], ('PAYMENT', 0.9948703646659851)], [[[428.0, 436.0], [470.0, 436.0], [470.0, 455.0], [428.0, 455.0]], ('9.05', 0.9961490631103516)], [[[152.0, 450.0], [251.0, 453.0], [250.0, 475.0], [152.0, 472.0]], ('Change Due', 0.9556287527084351)], [[[420.0, 458.0], [471.0, 458.0], [471.0, 480.0], [420.0, 480.0]], ('10.95', 0.997236430644989)], [[[209.0, 498.0], [382.0, 503.0], [381.0, 524.0], [208.0, 519.0]], ('$2.000FF', 0.9757758378982544)], [[[169.0, 522.0], [422.0, 528.0], [421.0, 548.0], [169.0, 542.0]], ('NEXT PURCHASE', 0.962527871131897)], [[[167.0, 546.0], [365.0, 552.0], [365.0, 570.0], [167.0, 564.0]], ('CALL800 705 5754or', 0.926964521408081)], [[[146.0, 570.0], [416.0, 577.0], [415.0, 597.0], [146.0, 590.0]], ('Go www.mshare.net/bajafresh', 0.9759786128997803)], [[[147.0, 594.0], [356.0, 601.0], [356.0, 621.0], [146.0, 614.0]], ('Take our brief survey', 0.9390400648117065)], [[[143.0, 620.0], [410.0, 626.0], [409.0, 647.0], [143.0, 641.0]], ('When Prompted, Enter Store', 0.9385656118392944)], [[[142.0, 646.0], [408.0, 653.0], [407.0, 673.0], [142.0, 666.0]], ('Write down redemption code', 0.9536812901496887)], [[[141.0, 672.0], [409.0, 679.0], [408.0, 699.0], [141.0, 692.0]], ('Use this receipt as coupon', 0.9658807516098022)], [[[138.0, 697.0], [448.0, 701.0], [448.0, 725.0], [138.0, 721.0]], ('Discount on purchases of $5.00', 0.9624248743057251)], [[[139.0, 726.0], [466.0, 729.0], [466.0, 750.0], [139.0, 747.0]], ('or more,Offer expires in 30 day', 0.9263916611671448)], [[[137.0, 750.0], [459.0, 755.0], [459.0, 778.0], [137.0, 773.0]], ('Good at participating locations', 0.963909924030304)]]

### 输出:
```json
{
  "receipt": {
    "store": "BAJA FRESH",
    "manager": "GENERAL MANAGER: NORMAN",
    "address": "176 Rosa C",
    "check": "Chk 7545",
    "date": "Dec26'0707:26PM",
    "tax": "Gst0",
    "total": "20.00",
    "payment": "CASH",
    "change": "0.61",
    "discount": "Discount on purchases of $5.00 or more,Offer expires in 30 day",
    "coupon": "Use this receipt as coupon",
    "survey": "Take our brief survey",
    "redemption": "Write down redemption code",
    "prompt": "When Prompted, Enter Store Write down redemption code Use this receipt as coupon",
    "items": [
      {
        "name": "1 BAJA STEAK",
        "price": "6.95",
        "modifiers": [
          "NO GUACAMOLE",
          "ENCHILADO STYLE"
        ]
      },
      {
        "name": "TAKE OUT",
        "price": "1.49"
      }
    ]
  }
}

寻找ChatML格式的模型？

请访问链接: https://huggingface.co/mychen76/mistral_ocr2json_v3_chatml

寻找收据图像转JSON的视觉模型

请访问链接: https://huggingface.co/mychen76/paligemma-receipt-json-3b-mix-448-v2b

直接加载模型

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mychen76/mistral7b_ocr_to_json_v1")
model = AutoModelForCausalLM.from_pretrained("mychen76/mistral7b_ocr_to_json_v1")

prompt=f"""### 指令:
您是POS收据数据专家，请解析、检测、识别并将以下收据OCR图像结果转换为结构化的收据数据对象。
不要编造输入中没有的值。输出必须是一个格式良好的JSON对象。```json

### 输入:
{receipt_boxes}

### 输出:
"""
with torch.inference_mode():
    inputs = tokenizer(prompt,return_tensors="pt",truncation=True).to(device)
    outputs = model.generate(**inputs, max_new_tokens=512) 
    result_text = tokenizer.batch_decode(outputs)[0]
    print(result_text)

获取OCR图像框

from paddleocr import PaddleOCR, draw_ocr
from ast import literal_eval
import json

paddleocr = PaddleOCR(lang="en",ocr_version="PP-OCRv4",show_log = False,use_gpu=True)

def paddle_scan(paddleocr,img_path_or_nparray):
    result = paddleocr.ocr(img_path_or_nparray,cls=True)
    result = result[0]
    boxes = [line[0] for line in result]       # 边界框
    txts = [line[1][0] for line in result]     # 原始文本
    scores = [line[1][1] for line in result]   # 分数
    return  txts, result

# 执行OCR扫描
receipt_texts, receipt_boxes = paddle_scan(paddleocr,receipt_image_array)
print(50*"--","\n仅文本:\