语言: 波斯语
数据集:

• shemo
  标签:
• 音频
• 自动语音识别
• 语音
• xlsr微调周
  许可证: apache-2.0
  示例:
• 标签: ShEMO样本250
  来源: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian-shemo/resolve/main/sample250.flac
• 标签: ShEMO样本52
  来源: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian-shemo/resolve/main/sample52.flac
  模型索引:
• 名称: Mehrdad Farahani的XLSR Wav2Vec2波斯语（Farsi）ShEMO
  结果:
  • 任务:
    名称: 语音识别
    类型: 自动语音识别
    数据集:
    名称: ShEMO fa
    类型: shemo
    参数: fa
    指标:
    • 名称: 测试WER
      类型: wer
      值: 30.00

Wav2Vec2-Large-XLSR-53-波斯语ShEMO

基于Wav2Vec2-Large-XLSR-53-波斯语V2在波斯语（Farsi）上使用ShEMO进行微调。使用此模型时，请确保语音输入采样率为16kHz。

使用方法

该模型可以直接使用（无需语言模型）如下：

环境要求

# 所需包  
!pip install git+https://github.com/huggingface/datasets.git  
!pip install git+https://github.com/huggingface/transformers.git  
!pip install torchaudio  
!pip install librosa  
!pip install jiwer  
!pip install hazm  
!pip install num2fawords  

预测代码

import librosa  
import torch  
import torchaudio  
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor  
from datasets import load_dataset  

from num2fawords import words, ordinal_words  
import numpy as np  
import hazm  
import re  
import string  

import IPython.display as ipd  

_normalizer = hazm.Normalizer()  

chars_to_ignore = [  
    ",", "?", ".", "!", "-", ";", ":", '""', "%", "'", '"', "�",  
    "#", "!", "؟", "?", "«", "»", "،", "(", ")", "؛", "'ٔ", "٬",'ٔ', ",", "?",   
    ".", "!", "-", ";", ":",'"',"“", "%", "‘", "”", "�", "–", "…", "_", "”", '“', '„',  
    'ā', 'š',  
    # "ء",   
]  

# 波斯语额外忽略字符  
chars_to_ignore = chars_to_ignore + list(string.ascii_lowercase + string.digits)  

chars_to_mapping = {  
    'ك': 'ک', 'دِ': 'د', 'بِ': 'ب', 'زِ': 'ز', 'ذِ': 'ذ', 'شِ': 'ش', 'سِ': 'س', 'ى': 'ی',  
    'ي': 'ی', 'أ': 'ا', 'ؤ': 'و', "ے": "ی", "ۀ": "ه", "ﭘ": "پ", "ﮐ": "ک", "ﯽ": "ی",  
    "ﺎ": "ا", "ﺑ": "ب", "ﺘ": "ت", "ﺧ": "خ", "ﺩ": "د", "ﺱ": "س", "ﻀ": "ض", "ﻌ": "ع",  
    "ﻟ": "ل", "ﻡ": "م", "ﻢ": "م", "ﻪ": "ه", "ﻮ": "و", 'ﺍ': "ا", 'ة': "ه",  
    'ﯾ': "ی", 'ﯿ': "ی", 'ﺒ': "ب", 'ﺖ': "ت", 'ﺪ': "د", 'ﺮ': "ر", 'ﺴ': "س", 'ﺷ': "ش",  
    'ﺸ': "ش", 'ﻋ': "ع", 'ﻤ': "م", 'ﻥ': "ن", 'ﻧ': "ن", 'ﻭ': "و", 'ﺭ': "ر", "ﮔ": "گ",  
        
    # "ها": "  ها", "ئ": "ی",  
        
    "a": " ای ", "b": " بی ", "c": " سی ", "d": " دی ", "e": " ایی ", "f": " اف ",  
    "g": " جی ", "h": "  اچ ", "i": " آی ", "j": " جی ", "k": " کی ", "l": " ال ",  
    "m": "  ام ", "n": " ان ", "o": " او ", "p": " پی ", "q": " کیو ", "r": " آر ",  
    "s": " اس ", "t": " تی ", "u": " یو ", "v": " وی ", "w": " دبلیو ", "x": " اکس ",  
    "y": "  وای ", "z": " زد ",  
    "\u200c": " ", "\u200d": " ", "\u200e": " ", "\u200f": " ", "\ufeff": " ",  
}  

def multiple_replace(text, chars_to_mapping):  
    pattern = "|".join(map(re.escape, chars_to_mapping.keys()))  
    return re.sub(pattern, lambda m: chars_to_mapping[m.group()], str(text))  

def remove_special_characters(text, chars_to_ignore_regex):  
    text = re.sub(chars_to_ignore_regex, '', text).lower() + " "  
    return text  

def normalizer(batch, chars_to_ignore, chars_to_mapping):  
    chars_to_ignore_regex = f"""[{"".join(chars_to_ignore)}]"""  
    text = batch["sentence"].lower().strip()  
    
    text = _normalizer.normalize(text)  
    text = multiple_replace(text, chars_to_mapping)  
    text = remove_special_characters(text, chars_to_ignore_regex)  
    text = re.sub(" +", " ", text)  
    _text = []  
    for word in text.split():  
        try:  
            word = int(word)  
            _text.append(words(word))  
        except:  
            _text.append(word)  
            
    text = " ".join(_text) + " "  
    
    text = text.strip() + " "  

    batch["sentence"] = text  
    return batch  


def speech_file_to_array_fn(batch):  
    speech_array, sampling_rate = torchaudio.load(batch["path"])  
    speech_array = speech_array.squeeze().numpy()  
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)  

    batch["speech"] = speech_array  
    return batch  


def predict(batch):  
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)  

    input_values = features.input_values.to(device)  
    attention_mask = features.attention_mask.to(device)  

    with torch.no_grad():  
        logits = model(input_values, attention_mask=attention_mask).logits   
        
    pred_ids = torch.argmax(logits, dim=-1)  

    batch["predicted"] = processor.batch_decode(pred_ids)[0]  
    return batch  


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian-shemo")  
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian-shemo").to(device)  

dataset = load_dataset("csv", data_files={"test": "/content/fa/dataset/test.csv"}, delimiter="\t")["test"]  
dataset = dataset.map(  
    normalizer,   
    fn_kwargs={"chars_to_ignore": chars_to_ignore, "chars_to_mapping": chars_to_mapping},  
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))  
)  

dataset = dataset.map(speech_file_to_array_fn)  
result = dataset.map(predict)  

max_items = np.random.randint(0, len(result), 20).tolist()  
for i in max_items:  
    reference, predicted =  result["sentence"][i], result["predicted"][i]  
    print("参考文本:", reference)  
    print("预测结果:", predicted)  
    print('---')  

输出示例:

参考文本: 那晚你发誓说爱我爱得胜过生命 承诺会陪我走到生命尽头 那晚你向我保证会为我举办最盛大的婚礼  
预测结果: 那晚你发誓说爱我爱得胜过生命 会陪我走到生命尽头 那晚你向我保证会为我举办最盛大的婚礼  
---  
参考文本: 你们整天骂他打他还不够吗  
预测结果: 你们整天骂他打他还不够吗  
---  
参考文本: 房子  
预测结果: 房子  
---  
参考文本: 别闹了  
预测结果: 别闹了  
---  
参考文本: 我给其他人预留了套房  
预测结果: 我给其他人预留了套房  
---  
参考文本: 滚出去 滚出去 滚出去  
预测结果: 滚出去 滚出去 滚出去  
---  
参考文本: 战争结束一年后你才明白真相  
预测结果: 战争结束一年后你才明白真相  
---  
参考文本: 除了那两个睡在这里的人 他没向你介绍其他朋友  
预测结果: 除了那两个睡在这里的人 他没向你介绍其他朋友  
---  
参考文本: 我知道你在这里 开门 快开门  
预测结果: 我知道你在这里 开门 快开门  
---  
参考文本: 作家需要有四只眼睛四只耳朵  
预测结果: 作家需要有四只眼睛四只耳朵  
---  

评估

可以在Common Voice的波斯语测试数据上评估模型如下：

import librosa  
import torch  
import torchaudio  
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor  
from datasets import load_dataset, load_metric  

from num2fawords import words, ordinal_words  
import numpy as np  
import hazm  
import re  
import string  

_normalizer = hazm.Normalizer()  

chars_to_ignore = [  
    ",", "?", ".", "!", "-", ";", ":", '""', "%", "'", '"', "�",  
    "#", "!", "؟", "?", "«", "»", "،", "(", ")", "؛", "'ٔ", "٬",'ٔ', ",", "?",   
    ".", "!", "-", ";", ":",'"',"“", "%", "‘", "”", "�", "–", "…", "_", "”", '“', '„',  
    'ā', 'š',  
    # "ء",   
]  

# 波斯语额外忽略字符  
chars_to_ignore = chars_to_ignore + list(string.ascii_lowercase + string.digits)  

chars_to_mapping = {  
    'ك': 'ک', 'دِ': 'د', 'بِ': 'ب', 'زِ': 'ز', 'ذِ': 'ذ', 'شِ': 'ش', 'سِ': 'س', 'ى': 'ی',  
    'ي': 'ی', 'أ': 'ا', 'ؤ': 'و', "ے": "ی", "ۀ": "ه", "ﭘ": "پ", "ﮐ": "ک", "ﯽ": "ی",  
    "ﺎ": "ا", "ﺑ": "ب", "ﺘ": "ت", "ﺧ": "خ", "ﺩ": "د", "ﺱ": "س", "ﻀ": "ض", "ﻌ": "ع",  
    "ﻟ": "ل", "ﻡ": "م", "ﻢ": "م", "ﻪ": "ه", "ﻮ": "و", 'ﺍ': "ا", 'ة': "ه",  
    'ﯾ': "ی", 'ﯿ': "ی", 'ﺒ': "ب", 'ﺖ': "ت", 'ﺪ': "د", 'ﺮ': "r", 'ﺴ': "س", 'ﺷ': "ش",  
    'ﺸ': "ش", 'ﻋ': "ع", 'ﻤ': "م", 'ﻥ': "ن", 'ﻧ': "n", 'ﻭ': "و", 'ﺭ': "r", "ﮔ": "گ",  
        
    # "ها": "  ها", "ئ": "ی",  
        
    "a": " ای ", "b": " بی ", "c": " سی ", "d": " دی ", "e": " ایی ", "f": " اف ",  
    "g": "  جی ", "h": "  اچ ", "i": "  آی ", "j": "  جی ", "k": "  کی ", "l": "  ال ",  
    "m": "  ام ", "n": "  ان ", "o": "  او ", "p": "  پی ", "q": "  کیو ", "r": "  آر ",  
    "s": "  اس ", "t": "  تی ", "u": "  یو ", "v": "  وی ", "w": "  دبلیو ", "x": "  اکس ",  
    "y": "  وای ", "z": "  زد ",  
    "\u200c": " ", "\u200d": " ", "\u200e": " ", "\u200f": " ", "\ufeff": " ",  
}  

def multiple_replace(text, chars_to_mapping):  
    pattern = "|".join(map(re.escape, chars_to_mapping.keys()))  
    return re.sub(pattern, lambda m: chars_to_mapping[m.group()], str(text))  

def remove_special_characters(text, chars_to_ignore_regex):  
    text = re.sub(chars_to_ignore_regex, '', text).lower() + " "  
    return text  

def normalizer(batch, chars_to_ignore, chars_to_mapping):  
    chars_to_ignore_regex = f"""[{"".join(chars_to_ignore)}]"""  
    text = batch["sentence"].lower().strip()  
    
    text = _normalizer.normalize(text)  
    text = multiple_replace(text, chars_to_mapping)  
    text = remove_special_characters(text, chars_to_ignore_regex)  
    text = re.sub(" +", " ", text)  
    _text = []  
    for word in text.split():  
        try:  
            word = int(word)  
            _text.append(words(word))  
        except:  
            _text.append(word)  
            
    text = " ".join(_text) + " "  
    
    text = text.strip() + " "  

    batch["sentence"] = text  
    return batch  


def speech_file_to_array_fn(batch):  
    speech_array, sampling_rate = torchaudio.load(batch["path"])  
    speech_array = speech_array.squeeze().numpy()  
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)  

    batch["speech"] = speech_array  
    return batch  


def predict(batch):  
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)  

    input_values = features.input_values.to(device)  
    attention_mask = features.attention_mask.to(device)  

    with torch.no_grad():  
        logits = model(input_values, attention_mask=attention_mask).logits   
        
    pred_ids = torch.argmax(logits, dim=-1)  

    batch["predicted"] = processor.batch_decode(pred_ids)[0]  
    return batch  


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian-shemo")  
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian-shemo").to(device)  

dataset = load_dataset("csv", data_files={"test": "/content/fa/dataset/test.csv"}, delimiter="\t")["test"]  
dataset = dataset.map(  
    normalizer,   
    fn_kwargs={"chars_to_ignore": chars_to_ignore, "chars_to_mapping": chars_to_mapping},  
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))  
)  
dataset = dataset.map(speech_file_to_array_fn)  
result = dataset.map(predict)  

wer = load_metric("wer")  
print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["sentence"])))  

测试结果:

• WER: 31.00%

训练过程

使用了Common Voice的train和validation数据集进行训练。

训练脚本可在此处找到。