许可证：Apache-2.0
语言：

• 英文
  任务标签：视频文本转文本
  标签：
• 视频
• 视频理解
• 视觉
• 多模态
• 对话式
• 自定义代码
• 指令调优
  库名称：transformers

Apollo：探索大型多模态模型中的视频理解

Apollo是一系列大型多模态模型（LMMs），致力于推动视频理解领域的最新技术发展。它支持以下任务：

• 长视频内容理解
• 时序推理
• 复杂视频问答
• 基于视频内容的多轮对话

Apollo模型擅长处理长达一小时的视频，通过策略性设计平衡速度与准确性。我们的模型仅需30亿参数即可超越多数70亿参数的竞品，甚至能与300亿规模的模型媲美。

核心亮点：

• 70亿参数变体
• 每帧32个标记

快速开始

安装：

pip install -e .  
pip install flash-attn --no-build-isolation  

推理示例：

import torch  
from transformers import AutoModelForCausalLM  
from apollo.mm_utils import (  
    KeywordsStoppingCriteria,  
    tokenizer_mm_token,  
    ApolloMMLoader  
)  
from apollo.conversations import conv_templates, SeparatorStyle  
from huggingface_hub import snapshot_download  

model_url = "Apollo-LMMs/Apollo-3B-t32"  
model_path = snapshot_download(model_url, repo_type="model")  

device = "cuda" if torch.cuda.is_available() else "cpu"  
model = AutoModelForCausalLM.from_pretrained(  
    model_path,  
    trust_remote_code=True,  
    low_cpu_mem_usage=True  
).to(device=device, dtype=torch.bfloat16)  

tokenizer = model.tokenizer  
vision_processors = model.vision_tower.vision_processor  
config = model.config  
num_repeat_token = config.mm_connector_cfg['num_output_tokens']  
mm_processor = ApolloMMLoader(  
    vision_processors,  
    config.clip_duration,  
    frames_per_clip=4,  
    clip_sampling_ratio=0.65,  
    model_max_length=config.model_max_length,  
    device=device,  
    num_repeat_token=num_repeat_token  
)  

video_path = "path/to/video.mp4"  
question = "详细描述这段视频内容"  
mm_data, replace_string = mm_processor.load_video(video_path)  

conv = conv_templates["qwen_2"].copy()  
conv.append_message(conv.roles[0], replace_string + "\n\n" + question)  
conv.append_message(conv.roles[1], None)  

prompt = conv.get_prompt()  
input_ids = tokenizer_mm_token(prompt, tokenizer, return_tensors="pt").unsqueeze(0).to(device)  

stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2  
stopping_criteria = KeywordsStoppingCriteria([stop_str], tokenizer, input_ids)  

with torch.inference_mode():  
    output_ids = model.generate(  
        input_ids,  
        vision_input=[mm_data],  
        data_types=['video'],  
        do_sample=True,  
        temperature=0.4,  
        max_new_tokens=256,  
        top_p=0.7,  
        use_cache=True,  
        num_beams=1,  
        stopping_criteria=[stopping_criteria]  
    )  

pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()  
print(pred)  

引用

如果您认为本项目有价值，请考虑引用：

@article{zohar2024apollo,  
    title={Apollo: An Exploration of Video Understanding in Large Multimodal Models},  
    author={Zohar, Orr and Wang, Xiaohan and Dubois, Yann and Mehta, Nikhil and Xiao, Tong and Hansen-Estruch, Philippe and Yu, Licheng and Wang, Xiaofang and Juefei-Xu, Felix and Zhang, Ning and Yeung-Levy, Serena and Xia, Xide},  
    journal={arXiv preprint arXiv:2412.10360},  
    year={2024}  
}  

更多详情，请访问项目网站或查阅论文。