语言:

• 英语
• 加泰罗尼亚语

数据集:

• covost2
• librispeech_asr

标签:

• 音频
• 语音翻译
• 自动语音识别
• 语音转文本2

许可证: mit 管道标签: 自动语音识别

示例:

• 示例标题: Common Voice 1 来源: https://cdn-media.huggingface.co/speech_samples/common_voice_en_18301577.mp3
• 示例标题: Common Voice 2 来源: https://cdn-media.huggingface.co/speech_samples/common_voice_en_99989.mp3
• 示例标题: Common Voice 3 来源: https://cdn-media.huggingface.co/speech_samples/common_voice_en_9999.mp3

S2T2-Wav2Vec2-CoVoST2-英语-加泰罗尼亚语-语音翻译模型

s2t-wav2vec2-large-en-ca 是一个专为端到端语音翻译（ST）训练的语音转文本Transformer模型。该S2T2模型由论文《大规模自监督与半监督学习在语音翻译中的应用》提出，并正式发布于Fairseq。

模型描述

S2T2是一个基于Transformer的序列到序列（语音编码器-解码器）模型，专为端到端自动语音识别（ASR）和语音翻译（ST）设计。它采用预训练的Wav2Vec2作为编码器，并搭配基于Transformer的解码器。模型通过标准的自回归交叉熵损失进行训练，并以自回归方式生成翻译结果。

预期用途与限制

此模型可用于端到端的英语语音到加泰罗尼亚语文本翻译。更多S2T2模型检查点，请访问模型中心。

使用方法

由于这是一个标准的序列到序列Transformer模型，您可以通过将语音特征传递给模型的generate方法来生成转录文本。

您可以直接通过ASR管道使用模型：

from datasets import load_dataset
from transformers import pipeline

librispeech_en = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
asr = pipeline("automatic-speech-recognition", model="facebook/s2t-wav2vec2-large-en-ca", feature_extractor="facebook/s2t-wav2vec2-large-en-ca")

translation = asr(librispeech_en[0]["file"])

或按步骤操作如下：

import torch
from transformers import Speech2Text2Processor, SpeechEncoderDecoder
from datasets import load_dataset

import soundfile as sf
model = SpeechEncoderDecoder.from_pretrained("facebook/s2t-wav2vec2-large-en-ca")
processor = Speech2Text2Processor.from_pretrained("facebook/s2t-wav2vec2-large-en-ca")

def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch
    
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"])
transcription = processor.batch_decode(generated_ids)

评估结果

CoVoST-V2测试集英语-加泰罗尼亚语翻译的BLEU得分：34.1

更多信息，请参阅官方论文中的表2第10行。

BibTeX引用信息

@article{DBLP:journals/corr/abs-2104-06678,
  author    = {Changhan Wang and
               Anne Wu and
               Juan Miguel Pino and
               Alexei Baevski and
               Michael Auli and
               Alexis Conneau},
  title     = {Large-Scale Self- and Semi-Supervised Learning for Speech Translation},
  journal   = {CoRR},
  volume    = {abs/2104.06678},
  year      = {2021},
  url       = {https://arxiv.org/abs/2104.06678},
  archivePrefix = {arXiv},
  eprint    = {2104.06678},
  timestamp = {Thu, 12 Aug 2021 15:37:06 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2104-06678.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}