许可协议：apache-2.0
标签：

• 图像描述生成
  支持语言：
• 英语
  任务类型：图像到文本
  数据集：
• michelecafagna26/hl-narratives
  语言：
• 英语
  评估指标：
• sacrebleu
• rouge
  库名称：transformers

基于GIT的叙事性图像描述微调模型

GIT基础模型在HL Narratives数据集上训练，用于生成高层次叙事性描述

模型微调过程 🏋️‍

• 训练周期：3轮
• 学习率：5e−5
• 优化器：Adam
• 精度：半精度（fp16）

测试集指标 🧾

| Cider  | SacreBLEU  | Rouge-L|  
|--------|------------|--------|  
| 75.78  |   11.11    |  27.61 |  

模型实战演示 🚀

import requests  
from PIL import Image  
from transformers import AutoProcessor, AutoModelForCausalLM  

processor = AutoProcessor.from_pretrained("git-base-captioning-ft-hl-narratives")  
model = AutoModelForCausalLM.from_pretrained("git-base-captioning-ft-hl-narratives").to("cuda")  

img_url = 'https://datasets-server.huggingface.co/assets/michelecafagna26/hl/--/default/train/0/image/image.jpg'   
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')  

inputs = processor(raw_image, return_tensors="pt").to("cuda")  
pixel_values = inputs.pixel_values  

generated_ids = model.generate(pixel_values=pixel_values, max_length=50,  
            do_sample=True,  
            top_k=120,  
            top_p=0.9,  
            early_stopping=True,  
            num_return_sequences=1)  

processor.batch_decode(generated_ids, skip_special_tokens=True)  

>>> "她正在海滩上摆姿势拍照，想发布到社交媒体上。"  

BibTex及引用信息

@inproceedings{cafagna2023hl,  
  title={{HL} {D}ataset: {V}isually-grounded {D}escription of {S}cenes, {A}ctions and  
{R}ationales},  
  author={Cafagna, Michele and van Deemter, Kees and Gatt, Albert},  
  booktitle={Proceedings of the 16th International Natural Language Generation Conference (INLG'23)},  
address = {Prague, Czech Republic},  
  year={2023}  
}