widget:

• src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png candidate_labels: 弹吉他的猫, 弹钢琴的狗 example_title: 吉他、猫和狗 language: zh license: mit

clip-vit-large-patch14-ko

基于《利用知识蒸馏实现单语语句嵌入的多语言化》训练的韩语CLIP模型

训练代码: https://github.com/Bing-su/KoCLIP_training_code

使用数据: AIHUB平台所有韩英平行语料

使用指南

方法一

import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo = "Bingsu/clip-vit-large-patch14-ko"
model = AutoModel.from_pretrained(repo)
processor = AutoProcessor.from_pretrained(repo)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=["两只猫", "两只狗"], images=image, return_tensors="pt", padding=True)
with torch.inference_mode():
    outputs = model(**inputs)
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)

>>> probs
tensor([[0.9974, 0.0026]])

方法二

from transformers import pipeline

repo = "Bingsu/clip-vit-large-patch14-ko"
pipe = pipeline("zero-shot-image-classification", model=repo)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
result = pipe(images=url, candidate_labels=["一只猫", "两只猫", "躺在粉色沙发上的猫伙伴们"], hypothesis_template="{}")

>>> result
[{'score': 0.9907576441764832, 'label': '躺在粉色沙发上的猫伙伴们'},
 {'score': 0.009206341579556465, 'label': '两只猫'},
 {'score': 3.606083555496298e-05, 'label': '一只猫'}]