library_name: transformers tags: []

基于MuAViC数据集的AV-HuBERT模型Huggingface实现

本代码库包含AV-HuBERT（视听隐藏单元BERT）模型的Huggingface实现，该模型专门在MuAViC（多语言视听语料库）数据集上进行了训练和测试。AV-HuBERT是一种自监督模型，专为视听语音识别设计，通过结合音频和视觉模态实现鲁棒性能，尤其在嘈杂环境中表现优异。

本代码库主要特性包括：

• 预训练模型：获取在MuAViC数据集上微调的预训练AV-HuBERT模型。预训练模型源自MuAViC代码库。

• 推理脚本：通过Huggingface接口轻松构建流程。

• 数据预处理脚本：包括帧率标准化、唇部与音频特征提取。

推理代码示例

git clone https://github.com/nguyenvulebinh/AV-HuBERT-S2S.git
cd AV-HuBERT-S2S
conda create -n avhuberts2s python=3.9
conda activate avhuberts2s
pip install -r requirements.txt
python run_example.py

from src.model.avhubert2text import AV2TextForConditionalGeneration
from src.dataset.load_data import load_feature
from transformers import Speech2TextTokenizer
import torch

if __name__ == "__main__":
    # 选择运行示例的语言
    AVAILABEL_LANGUAGES = ["ar", "de", "el", "en", "es", "fr", "it", "pt", "ru", "multilingual"]
    language = "ru"
    assert language in AVAILABEL_LANGUAGES, f"语言{language}不可用，请选择以下语言之一：{AVAILABEL_LANGUAGES}"
    
    
    # 加载模型和分词器
    model_name_or_path = f"nguyenvulebinh/AV-HuBERT-MuAViC-{language}"
    model = AV2TextForConditionalGeneration.from_pretrained(model_name_or_path, cache_dir='./model-bin')
    tokenizer = Speech2TextTokenizer.from_pretrained(model_name_or_path, cache_dir='./model-bin')
    
    model = model.cuda().eval()
    
    # 加载示例视频和音频
    video_example = f"./example/video_processed/{language}_lip_movement.mp4"
    audio_example = f"./example/video_processed/{language}_audio.wav"
    if not os.path.exists(video_example) or not os.path.exists(audio_example):
        print(f"警告：{language}的示例视频/音频不可用，将使用英语示例")
        video_example = f"./example/video_processed/en_lip_movement.mp4"
        audio_example = f"./example/video_processed/en_audio.wav"
    
    # 加载并处理示例
    sample = load_feature(
        video_example,
        audio_example
    )
    
    audio_feats = sample['audio_source'].cuda()
    video_feats = sample['video_source'].cuda()
    attention_mask = torch.BoolTensor(audio_feats.size(0), audio_feats.size(-1)).fill_(False).cuda()
    
    # 生成文本
    output = model.generate(
        audio_feats,
        attention_mask=attention_mask,
        video=video_feats,
        max_length=1024,
    )

    print(tokenizer.batch_decode(output, skip_special_tokens=True))

数据预处理脚本

mkdir model-bin
cd model-bin
wget https://huggingface.co/nguyenvulebinh/AV-HuBERT/resolve/main/20words_mean_face.npy .
wget https://huggingface.co/nguyenvulebinh/AV-HuBERT/resolve/main/shape_predictor_68_face_landmarks.dat .

# 当前仅支持4:3比例的原始视频
cp raw_video.mp4 ./example/ 

python src/dataset/video_to_audio_lips.py

预训练AVSR模型

致谢

AV-HuBERT：本代码库大部分实现基于原始AV-HuBERT项目。

MuAViC代码库：特别感谢MuAViC数据集和代码库的创建者提供本项目使用的预训练模型。

许可协议

CC-BY-NC 4.0

引用文献

@article{anwar2023muavic,
  title={MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation},
  author={Anwar, Mohamed and Shi, Bowen and Goswami, Vedanuj and Hsu, Wei-Ning and Pino, Juan and Wang, Changhan},
  journal={arXiv preprint arXiv:2303.00628},
  year={2023}
}