🚀 GenEx-World-Explorer 🌍

GenEx World Explorer 是一个基于 Stable Video Diffusion (SVD) 的视频生成管道。它以关键帧为输入，生成时间上连贯的视频。此探索者版本在 SVD 的基础上，采用了自定义的 UNetSpatioTemporalConditionModel。

该扩散器能够根据全景输入图像生成向前移动的路径，从而对给定场景进行探索。

🚀 快速开始

安装依赖

diffusers>=0.33.1
transformers
numpy
pillow

使用示例

from diffusers import UNetSpatioTemporalConditionModel, StableVideoDiffusionPipeline
import torch
from PIL import Image

model_id = 'genex-world/GenEx-World-Explorer'

# 加载自定义 UNet
unet = UNetSpatioTemporalConditionModel.from_pretrained(
    model_id,
    subfolder='unet',
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)

# 加载带有自定义 UNet 的完整管道
pipe = StableVideoDiffusionPipeline.from_pretrained(
    model_id,
    unet=unet,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    local_files_only=True,
).to('cuda')

# 探索世界！
image = Image.open('example.png').resize((1024, 576), Image.BICUBIC).convert('RGB')

generator = torch.manual_seed(-1)
with torch.inference_mode():
    frames = self.pipe(image,
                num_frames=25,
                width=1024,
                height=576,
                decode_chunk_size=8, generator=generator, motion_bucket_id=127, fps=7, num_inference_steps=30, noise_aug_strength=0.02).frames[0]

📚 详细文档

模型信息

引用信息

@misc{lu2025genexgeneratingexplorableworld,
      title={GenEx: Generating an Explorable World}, 
      author={Taiming Lu and Tianmin Shu and Junfei Xiao and Luoxin Ye and Jiahao Wang and Cheng Peng and Chen Wei and Daniel Khashabi and Rama Chellappa and Alan Yuille and Jieneng Chen},
      year={2025},
      eprint={2412.09624},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.09624}, 
}