# 视频多模态大语言模型的慢-快架构（Qwen2-7B，64帧）

本仓库包含论文《视频多模态大语言模型的慢-快架构》中提出的**慢-快视频MLLM模型（Qwen2-7B, ConvNeXt-576, 64帧, 步长1/4）**。

[代码仓库](https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM) | [HuggingFace合集](https://huggingface.co/collections/shi-labs/slow-fast-video-mllm-67ef347a28772734c15a78b5)

## 模型描述

该模型创新性地采用慢-快架构，解决了有限计算预算下视频多模态大语言模型（MLLMs）在时间分辨率和空间细节之间的平衡难题。现有方法常通过不可逆压缩丢失视频细节。

受人类先快速浏览视频再聚焦关键片段的行为启发，慢-快设计采用双token策略：
1. **"快"视觉token**：压缩后的紧凑视频特征集合，与文本嵌入共同输入LLM（Qwen2-7B-Instruct）实现快速概览
2. **"慢"视觉token**：通过特制混合解码层实现文本嵌入对原始视频特征的交叉注意力，以线性复杂度完成指令感知的视觉细节提取

该架构支持处理更多输入帧（如本检查点的64帧）同时保留空间细节，在视频理解基准测试中显著优于纯自注意力基线。本检查点采用Qwen2-7B-Instruct基座LLM和ConvNeXt-576视觉塔。

<div align="center">
<img src="https://huggingface.co/shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4/resolve/main/assets/images/fig-teaser.png" width="45%">
</div>

## 使用说明

**注意**：本模型依赖集成在`transformers`库中的自定义代码（`LlavaQwenSlowFastForCausalLM`）。请确保从[官方仓库](https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM)安装必要依赖，或加载模型时设置`trust_remote_code=True`。

本地运行时请先克隆仓库并安装依赖：
```bash
git clone https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM.git
cd Slow-Fast-Video-Multimodal-LLM
pip install --upgrade pip
pip install -r requirements.txt
# 将仓库路径添加至PYTHONPATH或直接安装

随后使用以下Python脚本：

import torch
import os
import numpy as np
from decord import VideoReader, cpu
import requests # 用于下载视频

# 确保可导入llava模块
# 若未从仓库安装，trust_remote_code=True会自动处理
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN
from llava.conversation import conv_templates
from llava.model.builder import load_pretrained_model
from llava.mm_utils import tokenizer_image_token, get_model_name_from_path
from llava.utils import disable_torch_init


def load_video(video_path, max_frames_num):
        """视频帧加载辅助函数"""
        vr = VideoReader(video_path, num_threads=4)
        total_frames = len(vr)

        # 确保稀疏采样不会导致帧数不足
        if total_frames >= max_frames_num:
            # 均匀采样视频帧
            uniform_sampled_frames = np.linspace(0, total_frames - 1, max_frames_num, dtype=int)
            frame_idx = uniform_sampled_frames.tolist()
        else:
            # 视频较短时采样所有帧并重复最后一帧
            frame_idx = list(range(total_frames))
            frame_idx.extend([total_frames - 1] * (max_frames_num - total_frames))

        try:
            spare_frames = vr.get_batch(frame_idx).asnumpy()
        except Exception as e:
            print(f"加载视频帧错误: {e}")
            # 错误处理：返回None或抛出异常
            raise e

        return spare_frames

# 模型配置
model_path = "shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4"
video_url = "https://huggingface.co/shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4/resolve/main/assets/catinterrupt.mp4"
video_local_path = "catinterrupt.mp4"
question = "请详细描述这个视频内容。"
max_frames = 64 # 本检查点训练使用64帧

# 下载示例视频
if not os.path.exists(video_local_path):
    print(f"从{video_url}下载视频...")
    response = requests.get(video_url, stream=True)
    response.raise_for_status()
    with open(video_local_path, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)
    print("下载完成")


# 加载模型和处理器
disable_torch_init()
model_name = get_model_name_from_path(model_path)

# 使用trust_remote_code加载自定义架构
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path,
    None,
    model_name,
    use_flash_attn=True,      # 启用Flash Attention
    device_map="auto",        # 自动分配GPU/CPU
    torch_dtype=torch.bfloat16, # 使用bfloat16提升效率
    trust_remote_code=True
)

# 构建提示词
if model.config.mm_use_im_start_end:
    prompt = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + "\n" + question
else:
    prompt = DEFAULT_IMAGE_TOKEN + "\n" + question

conv = conv_templates["qwen_1_5"].copy() # 使用适配的对话模板
conv.append_message(conv.roles[0], prompt)
conv.append_message(conv.roles[1], None)
prompt_final = conv.get_prompt()

# 加载处理视频帧
print("加载视频中...")
video_frames = load_video(video_local_path, max_frames_num=max_frames)
print(f"视频加载完成，形状: {video_frames.shape}")

# 预处理视频帧
print("视频预处理中...")
video_tensor = image_processor.preprocess(video_frames, return_tensors="pt")["pixel_values"]
video_tensor = video_tensor.to(model.device, dtype=torch.bfloat16)
videos = [video_tensor] # 模型需要视频张量列表
print(f"视频张量处理完成，形状: {videos[0].shape}")


# 分词处理提示词
input_ids = tokenizer_image_token(prompt_final, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt')
input_ids = input_ids.to(device=model.device, non_blocking=True)
if input_ids.ndim == 1:
    input_ids = input_ids.unsqueeze(0)
print(f"输入ID处理完成，形状: {input_ids.shape}")


# 生成响应
print("生成响应中...")
with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=videos, # 传入处理后的视频张量列表
        do_sample=True,
        temperature=0.2,
        top_p=1.0,
        num_beams=1,
        max_new_tokens=1024,
        use_cache=True
    )

# 解码输出
outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(f"\n用户输入: {question}\n")
print(f"模型输出:\n{outputs}")

许可协议

模型权重遵循CC-BY-NC-4.0许可。 代码部分采用Apache 2.0许可。 使用者须遵守包括基座语言模型(Qwen2许可)在内的所有原始许可条款。

引用

若使用本工作，请引用论文：

@misc{zhou2025slowfast,
      title={Slow-Fast Architecture for Video Multi-Modal Large Language Models},
      author={Yifei Zhou and Jiaming Zuo and Chen Change Loy and Chongyang Zhong and Xin Wang and Qi Wu and Weidong Cai and Xiaodong He and Qingzhong Wang and Lei Zhang and Marcelo H. Ang Jr and Boyang Li and Yanfeng Wang and Qinghai He and Fengbei Liu and Liangchen Luo and Jingdong Wang and Conghui He and Wenhai Wang},
      year={2025},
      eprint={2504.01328},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

(注：作者列表基于arXiv论文可能更新版本，请以最终发表版本为准)