BERTimbau基础版（又名"bert-base-portuguese-cased"）

简介

BERTimbau基础版是针对巴西葡萄牙语的预训练BERT模型，在命名实体识别、句子文本相似度和文本蕴含识别三个下游NLP任务中达到了最先进的性能表现。该模型提供两种规格：基础版和大规模版。

更多信息或需求，请访问BERTimbau项目仓库。

可用模型

使用方法

from transformers import AutoTokenizer  # 或BertTokenizer
from transformers import AutoModelForPreTraining  # 或BertForPreTraining用于加载预训练头
from transformers import AutoModel  # 或BertModel，用于无预训练头的BERT模型

model = AutoModelForPreTraining.from_pretrained('neuralmind/bert-base-portuguese-cased')
tokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-base-portuguese-cased', do_lower_case=False)

掩码语言建模预测示例

from transformers import pipeline

pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)

pipe('Tinha uma [MASK] no meio do caminho.')
# [{'score': 0.14287759363651276,
#  'sequence': '[CLS] Tinha uma pedra no meio do caminho. [SEP]',
#  'token': 5028,
#  'token_str': 'pedra'},
# {'score': 0.06213393807411194,
#  'sequence': '[CLS] Tinha uma árvore no meio do caminho. [SEP]',
#  'token': 7411,
#  'token_str': 'árvore'},
# {'score': 0.05515013635158539,
#  'sequence': '[CLS] Tinha uma estrada no meio do caminho. [SEP]',
#  'token': 5675,
#  'token_str': 'estrada'},
# {'score': 0.0299188531935215,
#  'sequence': '[CLS] Tinha uma casa no meio do caminho. [SEP]',
#  'token': 1105,
#  'token_str': 'casa'},
# {'score': 0.025660505518317223,
#  'sequence': '[CLS] Tinha uma cruz no meio do caminho. [SEP]',
#  'token': 3466,
#  'token_str': 'cruz'}]

获取BERT嵌入向量

import torch

model = AutoModel.from_pretrained('neuralmind/bert-base-portuguese-cased')
input_ids = tokenizer.encode('Tinha uma pedra no meio do caminho.', return_tensors='pt')

with torch.no_grad():
    outs = model(input_ids)
    encoded = outs[0][0, 1:-1]  # 忽略[CLS]和[SEP]特殊标记

# encoded.shape: (8, 768)
# tensor([[-0.0398, -0.3057,  0.2431,  ..., -0.5420,  0.1857, -0.5775],
#         [-0.2926, -0.1957,  0.7020,  ..., -0.2843,  0.0530, -0.4304],
#         [ 0.2463, -0.1467,  0.5496,  ...,  0.3781, -0.2325, -0.5469],
#         ...,
#         [ 0.0662,  0.7817,  0.3486,  ..., -0.4131, -0.2852, -0.2819],
#         [ 0.0662,  0.2845,  0.1871,  ..., -0.2542, -0.2933, -0.0661],
#         [ 0.2761, -0.1657,  0.3288,  ..., -0.2102,  0.0029, -0.2009]])

引用

如果使用我们的工作，请引用：

@inproceedings{souza2020bertimbau,
  author    = {F{\'a}bio Souza and
               Rodrigo Nogueira and
               Roberto Lotufo},
  title     = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
  booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
  year      = {2020}
}