Llave 2B
LLaVE-2B是基于Aquila-VL-2B模型的20亿参数多模态嵌入模型，具有4K tokens的上下文窗口，支持文本、图像、多图像和视频的嵌入表示。
下载量 20.05k
发布时间 : 2/9/2025
模型简介
LLaVE-2B是一个多模态嵌入模型，能够对文本、图像、多图像和视频进行嵌入表示，适用于句子相似度、零样本图像分类等任务。
模型特点
多模态嵌入
支持文本、图像、多图像和视频的嵌入表示，能够处理多种模态的数据。
4K tokens上下文窗口
具有4K tokens的上下文窗口，能够处理较长的输入序列。
零样本图像分类
能够在零样本设置下进行图像分类任务，无需额外训练数据。
迁移学习能力强
尽管在图像-文本数据上训练，但能泛化到文本-视频检索任务，表现出色。
模型能力
文本嵌入
图像嵌入
视频嵌入
多模态嵌入
句子相似度计算
零样本图像分类
视频-文本检索
使用案例
图像检索
图像-文本检索
根据文本描述检索相关图像
在MMEB排行榜上取得优异表现
视频检索
零样本视频-文本检索
根据文本描述检索相关视频
表现出色，展示了迁移到其他嵌入任务的潜力
🚀 LLaVE-2B
LLaVE-2B 是一个基于 Aquila-VL-2B 模型的 20 亿参数多模态嵌入模型，具有 4K 上下文窗口。它能够对文本、图像、多图像和视频进行嵌入处理，在 MMEB 排行榜上取得了优异成绩，且能零样本泛化到文本 - 视频检索任务。
🚀 快速开始
首先克隆我们的 GitHub 仓库：
git clone https://github.com/DeepLearnXMU/LLaVE
cd LLaVE
pip install -e ".[train]"
我们提供了使用该模型的简单嵌入过程。更多详细信息，请参考 Github。
# pip install git+https://github.com/DeepLearnXMU/LLaVE
import torch
import copy
from PIL import Image
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from llava.model.builder import load_pretrained_model
from llava.mm_utils import tokenizer_image_token, process_images
pretrained = "zhibinlan/LLaVE-2B"
model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map) # Add any other thing you want to pass in llava_model_args
model.eval()
# Image + Text -> Text
image = Image.open("figures/example.jpg")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
conv_template = "qwen_1_5" # Make sure you use correct chat template for different models
question = DEFAULT_IMAGE_TOKEN + " Represent the given image with the following question: What is in the image"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], "\n")
prompt_question = conv.get_prompt()
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
attention_mask=input_ids.ne(tokenizer.pad_token_id)
image_sizes = [image.size]
query_embed = model.encode_multimodal_embeddings(input_ids, attention_mask=attention_mask,images=image_tensor, image_sizes=image_sizes)
target_string = "A cat and a dog"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], target_string)
conv.append_message(conv.roles[1], "\n")
target_string = conv.get_prompt()
target_input_ids = tokenizer(target_string, return_tensors="pt").input_ids.to(device)
attention_mask=target_input_ids.ne(tokenizer.pad_token_id)
target_embed = model.encode_multimodal_embeddings(target_input_ids, attention_mask=attention_mask)
print("A cat and a dog similarity score: ", query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.5132]]
neg_string = "A cat and a tiger"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], neg_string)
conv.append_message(conv.roles[1], "\n")
neg_string = conv.get_prompt()
neg_input_ids = tokenizer(neg_string, return_tensors="pt").input_ids.to(device)
attention_mask=neg_input_ids.ne(tokenizer.pad_token_id)
neg_embed = model.encode_multimodal_embeddings(neg_input_ids, attention_mask=attention_mask)
print("A cat and a tiger similarity score: ", query_embed @ neg_embed.T)
# 2B: A cat and a tiger similarity score: tensor([[0.3809]]
# Text -> Image
pos_string = "Find me an everyday image that matches the given caption: A cat and a dog."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], pos_string)
conv.append_message(conv.roles[1], "\n")
pos_string = conv.get_prompt()
pos_input_ids = tokenizer(pos_string, return_tensors="pt").input_ids.to(device)
attention_mask=pos_input_ids.ne(tokenizer.pad_token_id)
pos_query_embed = model.encode_multimodal_embeddings(pos_input_ids, attention_mask=attention_mask)
target = DEFAULT_IMAGE_TOKEN + " Represent the given image."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], target)
conv.append_message(conv.roles[1], "\n")
prompt_target = conv.get_prompt()
target_input_ids = tokenizer_image_token(prompt_target, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
attention_mask=target_input_ids.ne(tokenizer.pad_token_id)
target_image_sizes = [image.size]
target_embed = model.encode_multimodal_embeddings(target_input_ids, attention_mask=attention_mask,images=image_tensor, image_sizes=target_image_sizes)
print("A cat and a dog image similarity score: ", pos_query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.5225]]
neg_string = "Find me an everyday image that matches the given caption: A cat and a tiger."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], neg_string)
conv.append_message(conv.roles[1], "\n")
neg_string = conv.get_prompt()
neg_input_ids = tokenizer(neg_string, return_tensors="pt").input_ids.to(device)
attention_mask=neg_input_ids.ne(tokenizer.pad_token_id)
neg_query_embed = model.encode_multimodal_embeddings(neg_input_ids, attention_mask=attention_mask)
print("A cat and a tiger image similarity score: ", neg_query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.4141]]
✨ 主要特性
- 多模态嵌入能力：能够对文本、图像、多图像和视频进行嵌入处理。
- 优异的排行榜成绩：在 MMEB 排行榜上仅使用少量数据就取得了顶尖排名。
- 零样本泛化：虽然在图像 - 文本数据上训练，但能零样本泛化到文本 - 视频检索任务，并取得良好性能。
📦 安装指南
git clone https://github.com/DeepLearnXMU/LLaVE
cd LLaVE
pip install -e ".[train]"
💻 使用示例
基础用法
# pip install git+https://github.com/DeepLearnXMU/LLaVE
import torch
import copy
from PIL import Image
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from llava.model.builder import load_pretrained_model
from llava.mm_utils import tokenizer_image_token, process_images
pretrained = "zhibinlan/LLaVE-2B"
model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map) # Add any other thing you want to pass in llava_model_args
model.eval()
# Image + Text -> Text
image = Image.open("figures/example.jpg")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
conv_template = "qwen_1_5" # Make sure you use correct chat template for different models
question = DEFAULT_IMAGE_TOKEN + " Represent the given image with the following question: What is in the image"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], "\n")
prompt_question = conv.get_prompt()
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
attention_mask=input_ids.ne(tokenizer.pad_token_id)
image_sizes = [image.size]
query_embed = model.encode_multimodal_embeddings(input_ids, attention_mask=attention_mask,images=image_tensor, image_sizes=image_sizes)
target_string = "A cat and a dog"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], target_string)
conv.append_message(conv.roles[1], "\n")
target_string = conv.get_prompt()
target_input_ids = tokenizer(target_string, return_tensors="pt").input_ids.to(device)
attention_mask=target_input_ids.ne(tokenizer.pad_token_id)
target_embed = model.encode_multimodal_embeddings(target_input_ids, attention_mask=attention_mask)
print("A cat and a dog similarity score: ", query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.5132]]
neg_string = "A cat and a tiger"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], neg_string)
conv.append_message(conv.roles[1], "\n")
neg_string = conv.get_prompt()
neg_input_ids = tokenizer(neg_string, return_tensors="pt").input_ids.to(device)
attention_mask=neg_input_ids.ne(tokenizer.pad_token_id)
neg_embed = model.encode_multimodal_embeddings(neg_input_ids, attention_mask=attention_mask)
print("A cat and a tiger similarity score: ", query_embed @ neg_embed.T)
# 2B: A cat and a tiger similarity score: tensor([[0.3809]]
高级用法
# Text -> Image
pos_string = "Find me an everyday image that matches the given caption: A cat and a dog."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], pos_string)
conv.append_message(conv.roles[1], "\n")
pos_string = conv.get_prompt()
pos_input_ids = tokenizer(pos_string, return_tensors="pt").input_ids.to(device)
attention_mask=pos_input_ids.ne(tokenizer.pad_token_id)
pos_query_embed = model.encode_multimodal_embeddings(pos_input_ids, attention_mask=attention_mask)
target = DEFAULT_IMAGE_TOKEN + " Represent the given image."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], target)
conv.append_message(conv.roles[1], "\n")
prompt_target = conv.get_prompt()
target_input_ids = tokenizer_image_token(prompt_target, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
attention_mask=target_input_ids.ne(tokenizer.pad_token_id)
target_image_sizes = [image.size]
target_embed = model.encode_multimodal_embeddings(target_input_ids, attention_mask=attention_mask,images=image_tensor, image_sizes=target_image_sizes)
print("A cat and a dog image similarity score: ", pos_query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.5225]]
neg_string = "Find me an everyday image that matches the given caption: A cat and a tiger."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], neg_string)
conv.append_message(conv.roles[1], "\n")
neg_string = conv.get_prompt()
neg_input_ids = tokenizer(neg_string, return_tensors="pt").input_ids.to(device)
attention_mask=neg_input_ids.ne(tokenizer.pad_token_id)
neg_query_embed = model.encode_multimodal_embeddings(neg_input_ids, attention_mask=attention_mask)
print("A cat and a tiger image similarity score: ", neg_query_embed @ target_embed.T)
# 2B: A cat and a dog similarity score: tensor([[0.4141]]
📚 详细文档
模型概述
LLaVE 模型是基于 Aquila-VL-2B 模型的 20 亿参数多模态嵌入模型，上下文窗口为 4K 个标记。
训练/评估数据
- 训练数据：https://huggingface.co/datasets/TIGER-Lab/MMEB-train
- 评估数据：https://huggingface.co/datasets/TIGER-Lab/MMEB-eval
预期用途
该模型具备对文本、图像、多图像和视频进行嵌入的能力。
MMEB 排行榜
我们仅使用少量数据就在 MMEB 排行榜上取得了顶尖排名。
模型性能
LLaVE-2B 使用较少的参数和 66.2 万个训练对在 MMEB 上取得了优异的性能。
尽管 LLaVE 是在图像 - 文本数据上训练的，但它可以零样本泛化到文本 - 视频检索任务，并取得强大的性能，展示了其在其他嵌入任务上的显著迁移潜力。
硬件与软件
- GPU：8 * Nvidia A100 (40G)（用于全模型训练）
- 编排工具：Huggingface Trainer
- 神经网络框架：PyTorch
🔧 技术细节
该模型基于 Aquila-VL-2B 模型构建，采用了硬度加权对比学习方法，能够对多模态数据进行有效的嵌入处理。
📄 许可证
本项目采用 Apache-2.0 许可证。
引用
@article{lan2025llave,
title={LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning},
author={Lan, Zhibin and Niu, Liqiang and Meng, Fandong and Zhou, Jie and Su, Jinsong},
journal={arXiv preprint arXiv:2503.04812},
year={2025}
}
Clip Vit Large Patch14 336
基于Vision Transformer架构的大规模视觉语言预训练模型，支持图像与文本的跨模态理解
文本生成图像
Transformers
TransformersC
openai
5.9M
241
Fashion Clip
MIT
FashionCLIP是基于CLIP开发的视觉语言模型，专门针对时尚领域进行微调，能够生成通用产品表征。
文本生成图像 Transformers 英语
Transformers 英语F
patrickjohncyh
3.8M
222
Gemma 3 1b It
Gemma 3是Google推出的轻量级先进开放模型系列，基于与Gemini模型相同的研究和技术构建。该模型是多模态模型，能够处理文本和图像输入并生成文本输出。
文本生成图像 Transformers
TransformersG
google
2.1M
347
Blip Vqa Base
Bsd-3-clause
BLIP是一个统一的视觉语言预训练框架，擅长视觉问答任务，通过语言-图像联合训练实现多模态理解与生成能力
文本生成图像 Transformers
TransformersB
Salesforce
1.9M
154
CLIP ViT H 14 Laion2b S32b B79k
MIT
基于OpenCLIP框架在LAION-2B英文数据集上训练的视觉-语言模型，支持零样本图像分类和跨模态检索任务
文本生成图像
Safetensors
SafetensorsC
laion
1.8M
368
CLIP ViT B 32 Laion2b S34b B79k
MIT
基于OpenCLIP框架在LAION-2B英语子集上训练的视觉-语言模型，支持零样本图像分类和跨模态检索
文本生成图像 Safetensors
SafetensorsC
laion
1.1M
112
Pickscore V1
PickScore v1 是一个针对文本生成图像的评分函数，可用于预测人类偏好、评估模型性能和图像排序等任务。
文本生成图像 Transformers
TransformersP
yuvalkirstain
1.1M
44
Owlv2 Base Patch16 Ensemble
Apache-2.0
OWLv2是一种零样本文本条件目标检测模型，可通过文本查询在图像中定位对象。
文本生成图像 Transformers
TransformersO
google
932.80k
99
Llama 3.2 11B Vision Instruct
Llama 3.2 是 Meta 发布的多语言多模态大型语言模型，支持图像文本到文本的转换任务，具备强大的跨模态理解能力。
文本生成图像 Transformers 支持多种语言
Transformers 支持多种语言L
meta-llama
784.19k
1,424
Owlvit Base Patch32
Apache-2.0
OWL-ViT是一个零样本文本条件目标检测模型，可以通过文本查询搜索图像中的对象，无需特定类别的训练数据。
文本生成图像 Transformers
TransformersO
google
764.95k
129
精选推荐AI模型
Llama 3 Typhoon V1.5x 8b Instruct
专为泰语设计的80亿参数指令模型，性能媲美GPT-3.5-turbo，优化了应用场景、检索增强生成、受限生成和推理任务
大型语言模型 Transformers 支持多种语言
Transformers 支持多种语言L
scb10x
3,269
16
Cadet Tiny
Openrail
Cadet-Tiny是一个基于SODA数据集训练的超小型对话模型，专为边缘设备推理设计，体积仅为Cosmo-3B模型的2%左右。
对话系统 Transformers 英语
Transformers 英语C
ToddGoldfarb
2,691
6
Roberta Base Chinese Extractive Qa
基于RoBERTa架构的中文抽取式问答模型，适用于从给定文本中提取答案的任务。
问答系统 中文
R
uer
2,694
98
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
智启未来，您的人工智能解决方案智库
简体中文