license: apache-2.0 language: fr library_name: transformers thumbnail: null tags:

• 自动语音识别
• hf-asr排行榜
• whisper-event datasets:
• mozilla-foundation/common_voice_11_0
• facebook/multilingual_librispeech
• facebook/voxpopuli
• google/fleurs
• gigant/african_accented_french metrics:
• 词错误率 model-index:
• name: 针对法语ASR微调的whisper-large-v2模型 results:
  • task: name: 自动语音识别 type: automatic-speech-recognition dataset: name: Common Voice 11.0 type: mozilla-foundation/common_voice_11_0 config: fr split: test args: fr metrics:
    • name: 贪婪解码词错误率 type: wer value: 8.15
    • name: 束搜索(beam=5)词错误率 type: wer value: 7.83
  • task: name: 自动语音识别 type: automatic-speech-recognition dataset: name: 多语言LibriSpeech(MLS) type: facebook/multilingual_librispeech config: french split: test args: french metrics:
    • name: 贪婪解码词错误率 type: wer value: 4.20
    • name: 束搜索(beam=5)词错误率 type: wer value: 4.03
  • task: name: 自动语音识别 type: automatic-speech-recognition dataset: name: VoxPopuli type: facebook/voxpopuli config: fr split: test args: fr metrics:
    • name: 贪婪解码词错误率 type: wer value: 9.10
    • name: 束搜索(beam=5)词错误率 type: wer value: 8.66
  • task: name: 自动语音识别 type: automatic-speech-recognition dataset: name: Fleurs type: google/fleurs config: fr_fr split: test args: fr_fr metrics:
    • name: 贪婪解码词错误率 type: wer value: 5.22
    • name: 束搜索(beam=5)词错误率 type: wer value: 4.98
  • task: name: 自动语音识别 type: automatic-speech-recognition dataset: name: 非洲口音法语 type: gigant/african_accented_french config: fr split: test args: fr metrics:
    • name: 贪婪解码词错误率 type: wer value: 4.58
    • name: 束搜索(beam=5)词错误率 type: wer value: 4.31

针对法语ASR微调的whisper-large-v2模型

本模型是基于openai/whisper-large-v2微调的版本，训练数据包含超过2200小时的法语语音音频，来自Common Voice 11.0、Multilingual LibriSpeech、Voxpopuli、Fleurs、Multilingual TEDx、MediaSpeech和African Accented French的训练集和验证集。使用模型时请确保语音输入采样率为16kHz。本模型不预测大小写或标点符号。

性能表现

下表是预训练模型在Common Voice 9.0、Multilingual LibriSpeech、Voxpopuli和Fleurs上的词错误率(WER)。这些结果来自原始论文。

下表是微调模型在Common Voice 11.0、Multilingual LibriSpeech、Voxpopuli和Fleurs上的词错误率。注意这些评估数据集已进行过滤和预处理，仅包含法语字母字符并移除了撇号外的标点符号。表格中结果格式为贪婪搜索WER / 束宽5的束搜索WER。

使用方法

使用🤗 Pipeline进行推理

import torch

from datasets import load_dataset
from transformers import pipeline

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# 加载pipeline
pipe = pipeline("automatic-speech-recognition", model="bofenghuang/whisper-large-v2-french", device=device)

# 注意：为生成工具设置forced_decoder_ids
pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(language="fr", task="transcribe")

# 加载数据
ds_mcv_test = load_dataset("mozilla-foundation/common_voice_11_0", "fr", split="test", streaming=True)
test_segment = next(iter(ds_mcv_test))
waveform = test_segment["audio"]

# 运行
generated_sentences = pipe(waveform, max_new_tokens=225)["text"]  # 贪婪解码
# generated_sentences = pipe(waveform, max_new_tokens=225, generate_kwargs={"num_beams": 5})["text"]  # 束搜索

# 如有必要对预测句子进行标准化处理

使用🤗底层API进行推理

import torch
import torchaudio

from datasets import load_dataset
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# 加载模型
model = AutoModelForSpeechSeq2Seq.from_pretrained("bofenghuang/whisper-large-v2-french").to(device)
processor = AutoProcessor.from_pretrained("bofenghuang/whisper-large-v2-french", language="french", task="transcribe")

# 注意：为生成工具设置forced_decoder_ids
model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(language="fr", task="transcribe")

# 16kHz
model_sample_rate = processor.feature_extractor.sampling_rate

# 加载数据
ds_mcv_test = load_dataset("mozilla-foundation/common_voice_11_0", "fr", split="test", streaming=True)
test_segment = next(iter(ds_mcv_test))
waveform = torch.from_numpy(test_segment["audio"]["array"])
sample_rate = test_segment["audio"]["sampling_rate"]

# 重采样
if sample_rate != model_sample_rate:
    resampler = torchaudio.transforms.Resample(sample_rate, model_sample_rate)
    waveform = resampler(waveform)

# 获取特征
inputs = processor(waveform, sampling_rate=model_sample_rate, return_tensors="pt")
input_features = inputs.input_features
input_features = input_features.to(device)

# 生成
generated_ids = model.generate(inputs=input_features, max_new_tokens=225)  # 贪婪解码
# generated_ids = model.generate(inputs=input_features, max_new_tokens=225, num_beams=5)  # 束搜索

# 解码
generated_sentences = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

# 如有必要对预测句子进行标准化处理