Bp Commonvoice100 Xlsr

由 lgris 开发

这是一个针对巴西葡萄牙语微调的Wav2vec 2.0模型，基于Common Voice 7.0数据集训练，支持葡萄牙语语音识别任务。

语音识别

Transformers

其他

开源协议:Apache-2.0 #巴西葡萄牙语语音识别 #Wav2Vec2微调 #多数据集评估

下载量 21

发布时间 : 3/2/2022

模型介绍

内容详情

替代品

模型简介

该模型是基于Wav2vec 2.0架构的自动语音识别(ASR)模型，专门针对巴西葡萄牙语进行微调，可用于将葡萄牙语语音转换为文本。

模型特点

多数据集评估

模型在多个葡萄牙语数据集上进行了全面评估，包括CETUC、Common Voice、LaPS BM等，确保广泛的适用性。

语言模型集成

支持与4-gram语言模型结合使用，显著降低词错误率(WER)，提高识别准确度。

高效推理

支持批量处理(batch processing)和GPU加速，适合实际应用场景部署。

模型能力

葡萄牙语语音识别

语音转文本

多数据集兼容

使用案例

语音转录

巴西葡萄牙语语音转录

将巴西葡萄牙语语音内容转换为文本

在CETUC数据集上WER为5.7%(使用语言模型)

教育技术

语言学习应用

用于开发葡萄牙语发音评估和语言学习工具

语言: 葡萄牙语
数据集:

common_voice
mls
cetuc
lapsbm
voxforge
tedx
sid
评估指标:
wer
标签:
音频
语音
wav2vec2
葡萄牙语
葡萄牙语语音语料库
自动语音识别
语音
PyTorch
许可证: apache-2.0

commonvoice100-xlsr: 基于Common Voice数据集的Wav2vec 2.0模型

这是一个针对巴西葡萄牙语微调的Wav2vec模型演示，使用了Common Voice 7.0数据集。

在本笔记本中，该模型将在其他可用的巴西葡萄牙语数据集上进行测试。

数据集	训练集	验证集	测试集
CETUC		--	5.4小时
Common Voice	37.8小时	--	9.5小时
LaPS BM		--	0.1小时
MLS		--	3.7小时
Multilingual TEDx (葡萄牙语)		--	1.8小时
SID		--	1.0小时
VoxForge		--	0.1小时
总计		--	21.6小时

摘要

	CETUC	CV	LaPS	MLS	SID	TEDx	VF	平均
commonvoice_100 (下方演示)	0.088	0.126	0.121	0.173	0.177	0.424	0.145	0.179
commonvoice_100 + 4-gram (下方演示)	0.057	0.095	0.076	0.138	0.146	0.382	0.130	0.146

演示

MODEL_NAME = "lgris/commonvoice100-xlsr"

导入和依赖

%%capture  
!pip install torch==1.8.2+cu111 torchvision==0.9.2+cu111 torchaudio===0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html  
!pip install datasets  
!pip install jiwer  
!pip install transformers  
!pip install soundfile  
!pip install pyctcdecode  
!pip install https://github.com/kpu/kenlm/archive/master.zip

import jiwer  
import torchaudio  
from datasets import load_dataset, load_metric  
from transformers import (  
    Wav2Vec2ForCTC,  
    Wav2Vec2Processor,  
)  
from pyctcdecode import build_ctcdecoder  
import torch  
import re  
import sys

辅助函数

chars_to_ignore_regex = '[\,\?\.\!\;\:\"]'  # noqa: W605  

def map_to_array(batch):  
    speech, _ = torchaudio.load(batch["path"])  
    batch["speech"] = speech.squeeze(0).numpy()   
    batch["sampling_rate"] = 16_000   
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower().replace("’", "'")  
    batch["target"] = batch["sentence"]  
    return batch

def calc_metrics(truths, hypos):  
    wers = []  
    mers = []  
    wils = []  
    for t, h in zip(truths, hypos):  
        try:  
            wers.append(jiwer.wer(t, h))  
            mers.append(jiwer.mer(t, h))  
            wils.append(jiwer.wil(t, h))  
        except: # 空字符串？  
            pass  
    wer = sum(wers)/len(wers)  
    mer = sum(mers)/len(mers)  
    wil = sum(wils)/len(wils)  
    return wer, mer, wil

def load_data(dataset):  
    data_files = {'test': f'{dataset}/test.csv'}  
    dataset = load_dataset('csv', data_files=data_files)["test"]  
    return dataset.map(map_to_array)

模型

class STT:  

    def __init__(self,   
                 model_name,   
                 device='cuda' if torch.cuda.is_available() else 'cpu',   
                 lm=None):  
        self.model_name = model_name  
        self.model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)  
        self.processor = Wav2Vec2Processor.from_pretrained(model_name)  
        self.vocab_dict = self.processor.tokenizer.get_vocab()  
        self.sorted_dict = {  
            k.lower(): v for k, v in sorted(self.vocab_dict.items(),   
                                            key=lambda item: item[1])  
        }  
        self.device = device  
        self.lm = lm  
        if self.lm:              
            self.lm_decoder = build_ctcdecoder(  
                list(self.sorted_dict.keys()),  
                self.lm  
            )  

    def batch_predict(self, batch):  
        features = self.processor(batch["speech"],   
                                  sampling_rate=batch["sampling_rate"][0],   
                                  padding=True,   
                                  return_tensors="pt")  
        input_values = features.input_values.to(self.device)  
        attention_mask = features.attention_mask.to(self.device)  
        with torch.no_grad():  
            logits = self.model(input_values, attention_mask=attention_mask).logits  
        if self.lm:  
            logits = logits.cpu().numpy()  
            batch["predicted"] = []  
            for sample_logits in logits:  
                batch["predicted"].append(self.lm_decoder.decode(sample_logits))  
        else:  
            pred_ids = torch.argmax(logits, dim=-1)  
            batch["predicted"] = self.processor.batch_decode(pred_ids)  
        return batch

下载数据集

%%capture  
!gdown --id 1HFECzIizf-bmkQRLiQD0QVqcGtOG5upI  
!mkdir bp_dataset  
!unzip bp_dataset -d bp_dataset/

测试

stt = STT(MODEL_NAME)

CETUC

ds = load_data('cetuc_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("CETUC WER:", wer)

CETUC WER: 0.08868880057404624

Common Voice

ds = load_data('commonvoice_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("CV WER:", wer)

CV WER: 0.12601035333655114

LaPS

ds = load_data('lapsbm_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("Laps WER:", wer)

Laps WER: 0.12149621212121209

MLS

ds = load_data('mls_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("MLS WER:", wer)

MLS WER: 0.173594387890256

SID

ds = load_data('sid_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("Sid WER:", wer)

Sid WER: 0.1775290775992294

TEDx

ds = load_data('tedx_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("TEDx WER:", wer)

TEDx WER: 0.4245704568241374

VoxForge

ds = load_data('voxforge_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("VoxForge WER:", wer)

VoxForge WER: 0.14541801948051947

使用语言模型的测试

# !find -type f -name "*.wav" -delete  
!rm -rf ~/.cache  
!gdown --id 1GJIKseP5ZkTbllQVgOL98R4yYAcIySFP  # 使用维基百科训练  
stt = STT(MODEL_NAME, lm='pt-BR-wiki.word.4-gram.arpa')  
# !gdown --id 1dLFldy7eguPtyJj5OAlI4Emnx0BpFywg  # 使用巴西葡萄牙语训练  
# stt = STT(MODEL_NAME, lm='pt-BR.word.4-gram.arpa')

CETUC

ds = load_data('cetuc_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("CETUC WER:", wer)

CETUC WER: 0.05764220069547976

Common Voice

ds = load_data('commonvoice_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("CV WER:", wer)

CV WER: 0.09569130510737103

LaPS

ds = load_data('lapsbm_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("Laps WER:", wer)

Laps WER: 0.07688131313131312

MLS

ds = load_data('mls_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("MLS WER:", wer)

MLS WER: 0.13814768877494732

SID

ds = load_data('sid_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("Sid WER:", wer)

Sid WER: 0.14652459944499036

TEDx

ds = load_data('tedx_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("TEDx WER:", wer)

TEDx WER: 0.38196090002435623

VoxForge

ds = load_data('voxforge_dataset')  
result = ds.map(stt.batch_predict, batched=True, batch_size=8)   
wer, mer, wil = calc_metrics(result["sentence"], result["predicted"])  
print("VoxForge WER:", wer)