language: pt license: mit tags:

• bert
• pytorch datasets:
• brWaC

BERTimbau Large (又名 "bert-large-portuguese-cased")

简介

BERTimbau Large 是一个针对巴西葡萄牙语的预训练 BERT 模型，在三个下游 NLP 任务上达到了最先进的性能：命名实体识别、句子文本相似度和文本蕴含识别。该模型提供两种规模：Base 和 Large。

更多信息或请求，请访问 BERTimbau 仓库。

可用模型

使用方法

from transformers import AutoTokenizer  # 或 BertTokenizer
from transformers import AutoModelForPreTraining  # 或 BertForPreTraining 用于加载预训练头部
from transformers import AutoModel  # 或 BertModel，用于不带预训练头部的 BERT

model = AutoModelForPreTraining.from_pretrained('neuralmind/bert-large-portuguese-cased')
tokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-large-portuguese-cased', do_lower_case=False)

掩码语言建模预测示例

from transformers import pipeline

pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)

pipe('Tinha uma [MASK] no meio do caminho.')
# [{'score': 0.5054386258125305,
#   'sequence': '[CLS] Tinha uma pedra no meio do caminho. [SEP]',
#   'token': 5028,
#   'token_str': 'pedra'},
#  {'score': 0.05616172030568123,
#   'sequence': '[CLS] Tinha uma curva no meio do caminho. [SEP]',
#   'token': 9562,
#   'token_str': 'curva'},
#  {'score': 0.02348282001912594,
#   'sequence': '[CLS] Tinha uma parada no meio do caminho. [SEP]',
#   'token': 6655,
#   'token_str': 'parada'},
#  {'score': 0.01795753836631775,
#   'sequence': '[CLS] Tinha uma mulher no meio do caminho. [SEP]',
#   'token': 2606,
#   'token_str': 'mulher'},
#  {'score': 0.015246033668518066,
#   'sequence': '[CLS] Tinha uma luz no meio do caminho. [SEP]',
#   'token': 3377,
#   'token_str': 'luz'}]

获取 BERT 嵌入

import torch

model = AutoModel.from_pretrained('neuralmind/bert-large-portuguese-cased')
input_ids = tokenizer.encode('Tinha uma pedra no meio do caminho.', return_tensors='pt')

with torch.no_grad():
    outs = model(input_ids)
    encoded = outs[0][0, 1:-1]  # 忽略 [CLS] 和 [SEP] 特殊标记

# encoded.shape: (8, 1024)
# tensor([[ 1.1872,  0.5606, -0.2264,  ...,  0.0117, -0.1618, -0.2286],
#         [ 1.3562,  0.1026,  0.1732,  ..., -0.3855, -0.0832, -0.1052],
#         [ 0.2988,  0.2528,  0.4431,  ...,  0.2684, -0.5584,  0.6524],
#         ...,
#         [ 0.3405, -0.0140, -0.0748,  ...,  0.6649, -0.8983,  0.5802],
#         [ 0.1011,  0.8782,  0.1545,  ..., -0.1768, -0.8880, -0.1095],
#         [ 0.7912,  0.9637, -0.3859,  ...,  0.2050, -0.1350,  0.0432]])

引用

如果您使用了我们的工作，请引用：

@inproceedings{souza2020bertimbau,
  author    = {F{\'a}bio Souza and
               Rodrigo Nogueira and
               Roberto Lotufo},
  title     = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
  booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
  year      = {2020}
}