许可证：cc-by-nc-4.0
库名称：fairseq
任务：文本转语音
标签：

• fairseq
• 音频
• 文本转语音
  语言：英语
  数据集：
• mtedx
• covost2
• europarl_st
• voxpopuli
  小部件示例：
• 示例标题：Common Voice样本1
  源音频：https://huggingface.co/facebook/xm_transformer_600m-es_en-multi_domain/resolve/main/common_voice_es_19966634.flac

unit_hifigan_mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj_dur

基于fairseq S2UT的语音到语音翻译模型（论文/代码）：

• 西班牙语到英语
• 训练数据：mTEDx、CoVoST 2、Europarl-ST和VoxPopuli

使用方法

import json  
import os  
from pathlib import Path  

import IPython.display as ipd  
from fairseq import hub_utils  
from fairseq.checkpoint_utils import load_model_ensemble_and_task_from_hf_hub  
from fairseq.models.speech_to_text.hub_interface import S2THubInterface  
from fairseq.models.text_to_speech import CodeHiFiGANVocoder  
from fairseq.models.text_to_speech.hub_interface import VocoderHubInterface  

from huggingface_hub import snapshot_download  
import torchaudio  

cache_dir = os.getenv("HUGGINGFACE_HUB_CACHE")  

#models, cfg, task = load_model_ensemble_and_task_from_hf_hub(  
#     "facebook/xm_transformer_s2ut_800m-es-en-st-asr-bt_h1_2022",  
#     arg_overrides={"config_yaml": "config.yaml", "task": "speech_to_text"},  
#     cache_dir=cache_dir,  
# )  
# model = models[0].cpu()  
# cfg["task"].cpu = True  
# generator = task.build_generator([model], cfg)  


# # 需要16kHz单声道音频  
# audio, _ = torchaudio.load("/Users/lpw/git/api-inference-community/docker_images/fairseq/tests/samples/sample2.flac")  

# sample = S2THubInterface.get_model_input(task, audio)  
# unit = S2THubInterface.get_prediction(task, model, generator, sample)  

# 语音合成           
library_name = "fairseq"  
cache_dir = (  
    cache_dir or (Path.home() / ".cache" / library_name).as_posix()  
)  
cache_dir = snapshot_download(  
    f"facebook/unit_hifigan_mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj_dur", cache_dir=cache_dir, library_name=library_name  
)  

x = hub_utils.from_pretrained(  
    cache_dir,  
    "model.pt",  
    ".",  
    archive_map=CodeHiFiGANVocoder.hub_models(),  
    config_yaml="config.json",  
    fp16=False,  
    is_vocoder=True,  
)  

with open(f"{x['args']['data']}/config.json") as f:  
    vocoder_cfg = json.load(f)  
assert (  
    len(x["args"]["model_path"]) == 1  
), "输入中包含过多的声码器模型"  

vocoder = CodeHiFiGANVocoder(x["args"]["model_path"][0], vocoder_cfg)  
tts_model = VocoderHubInterface(vocoder_cfg, vocoder)  

tts_sample = tts_model.get_model_input(unit)  
wav, sr = tts_model.get_prediction(tts_sample)  

ipd.Audio(wav, rate=sr)