language:

• bn licenses:
• cc-by-nc-sa-4.0

BanglaT5模型

本仓库包含BanglaT5模型的预训练检查点。这是一个基于"Span Corruption"目标预训练的序列到序列变换器模型。使用该检查点进行微调的模型在孟加拉语多项自然语言生成任务中达到了最先进的性能表现。

如需在不同下游任务（如机器翻译、抽象文本摘要、问答系统等）上进行微调，请参考官方GitHub仓库中的脚本。

注意：本模型使用特定文本规范化流程进行预训练，该流程可在此处获取。官方GitHub仓库中的所有微调脚本默认使用此规范化处理。若需将预训练模型适配其他任务，请确保在分词前通过该流程对文本单元进行规范化以获得最佳效果。基础示例如下：

在transformers中使用本模型（测试版本4.11.0.dev0）

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from normalizer import normalize # 通过pip install git+https://github.com/csebuetnlp/normalizer安装

model = AutoModelForSeq2SeqLM.from_pretrained("csebuetnlp/banglat5")
tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglat5", use_fast=False)

input_sentence = ""
input_ids = tokenizer(normalize(input_sentence), return_tensors="pt").input_ids
generated_tokens = model.generate(input_ids)
decoded_tokens = tokenizer.batch_decode(generated_tokens)[0]

print(decoded_tokens)

基准测试

• 有监督微调表现

基准测试数据集如下：

• MT: 机器翻译数据集
• TS: 抽象文本摘要数据集
• QA: 问答数据集
• MD: 多轮对话生成数据集
• NHG: 新闻标题生成数据集
• XLS: 跨语言摘要数据集

引用文献

若使用本模型，请引用以下论文：

@article{bhattacharjee2022banglanlg,
  author    = {Abhik Bhattacharjee and Tahmid Hasan and Wasi Uddin Ahmad and Rifat Shahriyar},
  title     = {BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla},
  journal   = {CoRR},
  volume    = {abs/2205.11081},
  year      = {2022},
  url       = {https://arxiv.org/abs/2205.11081},
  eprinttype = {arXiv},
  eprint    = {2205.11081}
}

若使用文本规范化模块，请引用以下论文：

@inproceedings{hasan-etal-2020-low,
    title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
    author = "Hasan, Tahmid  and
      Bhattacharjee, Abhik  and
      Samin, Kazi  and
      Hasan, Masum  and
      Basak, Madhusudan  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
    doi = "10.18653/v1/2020.emnlp-main.207",
    pages = "2612--2623",
    abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
}