🚀 首个自动文本到视频提示补全模型

本模型能够以几个单词作为输入，生成完整的文本到视频提示。

🚀 快速开始

本模型基于 Mistral-7B-v0.1，在 VidProM 数据集上进行微调，并使用了 8 块 A100 GPU。

📦 安装指南

下载模型

from transformers import pipeline
import torch
pipe = pipeline("text-generation", model="WenhaoWang/AutoT2VPrompt", model_kwargs={"torch_dtype": torch.bfloat16}, device_map="cuda:0")

设置参数

input = "An underwater world"      # 用于生成文本到视频提示的输入文本。
max_length = 50                    # 生成文本的最大长度。
temperature = 1.2                  # 控制生成的随机性。值越高，输出越随机。
top_k = 8                          # 在每一步将考虑的单词数量限制为最可能的前 k 个单词。
num_return_sequences = 10          # 从相同输入生成的不同文本到视频提示的数量。

生成提示

all_prompts = pipe(input, max_length = max_length, do_sample = True, temperature = temperature, top_k = top_k, num_return_sequences=num_return_sequences)

def process(text):
    text = text.replace('\n', '.')
    text = text.replace('  .', '.')
    text = text[:text.rfind('.')]
    text = text + '.'
    return text

for i in range(num_return_sequences):
    print(process(all_prompts[i]['generated_text']))

运行上述代码后，你将得到 10 条文本到视频提示，从中挑选你最喜欢的一条即可。

以下是生成的示例提示：

An underwater world, 25 ye boy, with aqua-green eyes, dk sandy blond hair, from the back, and on his back a fish, 23 ye old, weing glasses,ctoon chacte.
An underwater world, the video should capture the essence of tranquility and the beauty of nature.. a woman with short hair weing a green dress sitting at the desk.
An underwater world, the ocean is full of discded items, the water flows, and the light penetrating through the water.
An underwater world.. a woman with red eyes and red lips  is looking forwd.
An underwater world.. an old man sitting in a chair, smoking a pipe, a little smoke coming out of the chair, a man is drinking a glass.
An underwater world. The ocean is filled with bioluminess as the water reflects a soft glow from a bioluminescent phosphorescent light source. The camera slowly moves away and zooms in..
An underwater world. the girl looks at the camera and smiles with happiness..
An underwater world, 1960s horror film..
An underwater world.. 4 men in 1940s style clothes walk ound a gothic castle. night, fe. A girl is running, and there e some flowers along the river.
An underwater world,  -camera pan up . A girl is playing with her cat on a sunny day in the pk. A man is running and then falling down and dying.

📄 许可证

本模型采用 CC BY-NC 4.0 许可证。

📚 引用信息

@article{wang2024vidprom,
  title={VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models},
  author={Wang, Wenhao and Yang, Yi},
  journal={arXiv preprint arXiv:2403.06098},
  year={2024}
}

🙏 致谢

微调过程得到了 Yaowei Zheng 的帮助。

📞 联系我们

如果您有任何问题，请随时联系 Wenhao Wang (wangwenhao0716@gmail.com)。