license: mit
language:

• en
  base_model:
• THUDM/CogVideoX-2b
• Fudan-FUXI/LiFT-Critic-40b-lora
  pipeline_tag: text-to-video

LiFT：利用人类反馈实现文本到视频模型对齐

CogVideoX-1.5-LiFT 是基于奖励加权学习方法对 CogVideoX-1.5 进行微调的版本。

🚀 快速开始

我们提供了 cli_demo.py 供用户快速上手。

import argparse  
from typing import Literal  

import torch  
from diffusers import (  
    CogVideoXPipeline,  
    CogVideoXDDIMScheduler,  
    CogVideoXDPMScheduler,  
)  
from diffusers.utils import export_to_video, load_image, load_video  

def generate_video(  
    prompt: str,  
    model_path: str,  
    lora_path: str = None,  
    lora_rank: int = 128,  
    output_path: str = "./output.mp4",  
    image_or_video_path: str = "",  
    num_inference_steps: int = 50,  
    guidance_scale: float = 6.0,  
    num_videos_per_prompt: int = 1,  
    dtype: torch.dtype = torch.bfloat16,  
    generate_type: str = Literal["t2v", "i2v", "v2v"],   
    seed: int = 42,  
):  

    pipe = CogVideoXPipeline.from_pretrained(model_path, torch_dtype=dtype)  

    if lora_path:  
        pipe.load_lora_weights(lora_path, weight_name="pytorch_lora_weights.safetensors", adapter_name="test")  
        pipe.fuse_lora(lora_scale=1 / lora_rank, components=['transformer'])  

    pipe.scheduler = CogVideoXDPMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")  

    pipe.to("cuda")  

    video_generate = pipe(  
        prompt=prompt,  
        num_videos_per_prompt=num_videos_per_prompt,  
        num_inference_steps=num_inference_steps,  
        num_frames=49,  
        use_dynamic_cfg=True,  
        guidance_scale=guidance_scale,  
        generator=torch.Generator().manual_seed(seed),  
    ).frames[0]  

    export_to_video(video_generate, output_path, fps=8)  

if __name__ == "__main__":  
    parser = argparse.ArgumentParser(description="Generate a video from a text prompt using CogVideoX")  
    parser.add_argument(  
        "--model_path", type=str, default='Fudan-FUXI/CogVideoX-2B-LiFT', help="预训练模型的路径"  
    )  
    parser.add_argument(  
        "--prompt", type=str, default="A girl riding a bike.", help="生成视频的描述"  
    )  
    parser.add_argument(  
        "--output_path", type=str, default="./output.mp4", help="生成视频的保存路径"  
    )  
    parser.add_argument(  
        "--num_inference_steps", type=int, default=50, help="推理过程的步数"  
    )  
    parser.add_argument(  
        "--dtype", type=str, default="float16", help="计算数据类型（如 'float16' 或 'bfloat16'）"  
    )  
    parser.add_argument("--seed", type=int, default=42, help="随机种子以确保可复现性")  

    args = parser.parse_args()  
    dtype = torch.float16 if args.dtype == "float16" else torch.bfloat16  
    generate_video(  
        prompt=args.prompt,  
        model_path=args.model_path,  
        output_path=args.output_path,  
        num_inference_steps=args.num_inference_steps,  
        dtype=dtype,  
        generate_type='t2v',  
        seed=args.seed,  
    )  

运行脚本：

$ python cli_demo.py --prompt "a girl riding a bike." --model_path Fudan-FUXI/CogVideoX-2B-LiFT  

🖊️ 引用

如果您觉得我们的工作有帮助，请引用我们的论文。

@article{LiFT,  
  title={LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment.},  
  author={Wang, Yibin and Tan, Zhiyu, and Wang, Junyan and Yang, Xiaomeng and Jin, Cheng and Li, Hao},  
  journal={arXiv preprint arXiv:2412.04814},  
  year={2024}  
}