language: pt datasets:

• common_voice
• mls
• cetuc
• lapsbm
• voxforge
• tedx
• sid metrics:
• wer tags:
• audio
• speech
• wav2vec2
• pt
• portuguese-speech-corpus
• automatic-speech-recognition
• speech
• PyTorch
• hf-asr-leaderboard model-index:
• name: bp500-base10k_voxpopuli results:
  • task: name: Automatic Speech Recognition type: automatic-speech-recognition dataset: name: Common Voice type: common_voice args: pt metrics:
    • name: Test WER type: wer value: 24.9 license: apache-2.0

bp500-base10k_voxpopuli：基于巴西葡萄牙语（BP）数据集的Wav2vec 2.0模型

这是一个针对巴西葡萄牙语进行微调的Wav2vec模型演示，使用了以下数据集：

• CETUC：包含约145小时的巴西葡萄牙语语音，由50名男性和50名女性发音人录制，每人朗读约1,000个从CETEN-Folha语料库中选出的语音平衡句子。
• Common Voice 7.0：由Mozilla基金会提出的项目，旨在创建多种语言的开放数据集。志愿者通过官方网站捐赠并验证语音。
• Lapsbm：“Falabrasil - UFPA”是Fala Brasil小组用于基准测试巴西葡萄牙语ASR系统的数据集。包含35名发音人（10名女性），每人朗读20个独特句子，总计700条巴西葡萄牙语语音。音频以22.05 kHz采样率录制，无环境控制。
• Multilingual Librispeech (MLS)：一个包含多种语言的大规模数据集，基于LibriVox等公共领域的有声书录音。葡萄牙语子集（主要为巴西变体）包含约284小时语音，来自62名发音人朗读的55本有声书。
• Multilingual TEDx：包含8种语言TEDx演讲的音频集。葡萄牙语子集（主要为巴西变体）含164小时转录语音。
• Sidney (SID)：包含72名发音人（20名女性）的5,777条录音，年龄17至59岁，涵盖出生地、年龄、性别、教育及职业等信息。
• VoxForge：旨在构建开放声学模型数据集的项目。其巴西葡萄牙语语料包含约100名发音人和4,130条语音，采样率从16kHz至44.1kHz不等。

这些数据集被合并为更大的巴西葡萄牙语数据集。除Common Voice的开发/测试集用于验证/测试外，其余数据均用于训练。我们还为所有数据集创建了测试集。

原始模型使用fairseq微调。本笔记本使用转换后的版本，原始fairseq模型链接在此。

性能摘要

转录示例

演示

MODEL_NAME = "lgris/bp500-base10k_voxpopuli" 

导入与依赖

%%capture
!pip install torch==1.8.2+cu111 torchvision==0.9.2+cu111 torchaudio===0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
!pip install datasets
!pip install jiwer
!pip install transformers
!pip install soundfile
!pip install pyctcdecode
!pip install https://github.com/kpu/kenlm/archive/master.zip

import jiwer
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
    Wav2Vec2ForCTC,
    Wav2Vec2Processor,
)
from pyctcdecode import build_ctcdecoder
import torch
import re
import sys

辅助函数

chars_to_ignore_regex = '[\,\?\.\!\;\:\"]'  # 忽略特殊字符

def map_to_array(batch):
    speech, _ = torchaudio.load(batch["path"])
    batch["speech"] = speech.squeeze(0).numpy() 
    batch["sampling_rate"] = 16_000 
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")
    batch["target"] = batch["sentence"]
    return batch

def calc_metrics(truths, hypos):
    wers = []
    mers = []
    wils = []
    for t, h in zip(truths, hypos):
        try:
            wers.append(jiwer.wer(t, h))
            mers.append(jiwer.mer(t, h))
            wils.append(jiwer.wil(t, h))
        except: # 处理空字符串
            pass
    wer = sum(wers)/len(wers)
    mer = sum(mers)/len(mers)
    wil = sum(wils)/len(wils)
    return wer, mer, wil

def load_data(dataset):
    data_files = {'test': f'{dataset}/test.csv'}
    dataset = load_dataset('csv', data_files=data_files)["test"]
    return dataset.map(map_to_array)

模型类

class STT:

    def __init__(self, 
                 model_name, 
                 device='cuda' if torch.cuda.is_available() else 'cpu', 
                 lm=None):
        self.model_name = model_name
        self.model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
        self.processor = Wav2Vec2Processor.from_pretrained(model_name)
        self.vocab_dict = self.processor.tokenizer.get_vocab()
        self.sorted_dict = {
            k.lower(): v for k, v in sorted(self.vocab_dict.items(), 
                                            key=lambda item: item[1])
        }
        self.device = device
        self.lm = lm
        if self.lm:            
            self.lm_decoder = build_ctcdecoder(
                list(self.sorted_dict.keys()),
                self.lm
            )

    def batch_predict(self, batch):
        features = self.processor(batch["speech"], 
                                  sampling_rate=batch["sampling_rate"][0], 
                                  padding=True, 
                                  return_tensors="pt")
        input_values = features.input_values.to(self.device)
        with torch.no_grad():
            logits = self.model(input_values).logits
        if self.lm:
            logits = logits.cpu().numpy()
            batch["predicted"] = []
            for sample_logits in logits:
                batch["predicted"].append(self.lm_decoder.decode(sample_logits))
        else:
            pred_ids = torch.argmax(logits, dim=-1)
            batch["predicted"] = self.processor.batch_decode(pred_ids)
        return batch

下载数据集

%%capture
!gdown --id 1HFECzIizf-bmkQRLiQD0QVqcGtOG5upI
!mkdir bp_dataset
!unzip bp_dataset -d bp_dataset/

%cd bp_dataset 

/content/bp_dataset

测试

基础模型测试

stt = STT(MODEL_NAME)

CETUC数据集

ds = load_data('cetuc_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("CETUC WER:", wer)

CETUC WER: 0.12096759949218888

Common Voice数据集

ds = load_data('commonvoice_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("CV WER:", wer)

CV WER: 0.24977003159495725

LaPS数据集

ds = load_data('lapsbm_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("Laps WER:", wer)

Laps WER: 0.039769570707070705

MLS数据集

ds = load_data('mls_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("MLS WER:", wer)

MLS WER: 0.2269637077788063

SID数据集

ds = load_data('sid_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("Sid WER:", wer)

Sid WER: 0.1691680138494731

TEDx数据集

ds = load_data('tedx_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("TEDx WER:", wer)

TEDx WER: 0.34908555859018014

VoxForge数据集

ds = load_data('voxforge_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("VoxForge WER:", wer)

VoxForge WER: 0.11649350649350651

语言模型增强测试

!rm -rf ~/.cache
!gdown --id 1GJIKseP5ZkTbllQVgOL98R4yYAcIySFP  # 使用维基百科训练的4-gram模型
stt = STT(MODEL_NAME, lm='pt-BR-wiki.word.4-gram.arpa')

CETUC数据集（带LM）

ds = load_data('cetuc_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("CETUC WER:", wer)

CETUC WER: 0.07499558425787961

Common Voice数据集（带LM）

ds = load_data('commonvoice_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("CV WER:", wer)

CV WER: 0.17442648452610307

LaPS数据集（带LM）

ds = load_data('lapsbm_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("Laps WER:", wer)

Laps WER: 0.032774621212121206

MLS数据集（带LM）

ds = load_data('mls_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("MLS WER:", wer)

MLS WER: 0.18213620321569274

SID数据集（带LM）

ds = load_data('sid_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("Sid WER:", wer)

Sid WER: 0.18102544972868206

TEDx数据集（带LM）

ds = load_data('tedx_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("TEDx WER:", wer)

TEDx WER: 0.3491402028105601

VoxForge数据集（带LM）

ds = load_data('voxforge_dataset')
result = ds.map(stt.batch_predict, batched=True, batch_size=8) 
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])
print("VoxForge WER:", wer)

VoxForge WER: 0.11189529220779222