许可协议：Apache-2.0
标签：

• 图像描述生成
  支持语言：
• 英语
  任务类型：图像转文本
  数据集：
• michelecafagna26/hl
  语言：
• 英语
  评估指标：
• sacrebleu
• rouge
  库名称：transformers

基于BLIP-base的模型针对场景高层描述进行图像描述生成微调

BLIP基础模型在HL数据集上训练，用于图像场景生成

模型微调过程 🏋️‍

• 训练10个周期
• 学习率：5e−5
• 使用Adam优化器
• 半精度（fp16）训练

测试集评估指标 🧾

| Cider  | SacreBLEU  | Rouge-L |  
|--------|------------|---------|  
| 116.70 |   26.46    |  35.30  |  

模型实际应用示例 🚀

import requests  
from PIL import Image  
from transformers import BlipProcessor, BlipForConditionalGeneration  

processor = BlipProcessor.from_pretrained("michelecafagna26/blip-base-captioning-ft-hl-scenes")  
model = BlipForConditionalGeneration.from_pretrained("michelecafagna26/blip-base-captioning-ft-hl-scenes").to("cuda")  

img_url = 'https://datasets-server.huggingface.co/assets/michelecafagna26/hl/--/default/train/0/image/image.jpg'   
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')  

inputs = processor(raw_image, return_tensors="pt").to("cuda")  
pixel_values = inputs.pixel_values  

generated_ids = model.generate(pixel_values=pixel_values, max_length=50,  
            do_sample=True,  
            top_k=120,  
            top_p=0.9,  
            early_stopping=True,  
            num_return_sequences=1)  

processor.batch_decode(generated_ids, skip_special_tokens=True)  

>>> "照片拍摄于湖泊附近。"  

BibTex及引用信息

@inproceedings{cafagna2023hl,  
  title={{HL} {D}ataset: {V}isually-grounded {D}escription of {S}cenes, {A}ctions and  
{R}ationales},  
  author={Cafagna, Michele and van Deemter, Kees and Gatt, Albert},  
  booktitle={Proceedings of the 16th International Natural Language Generation Conference (INLG'23)},  
address = {Prague, Czech Republic},  
  year={2023}  
}