language: ja
license: cc-by-sa-4.0
tags:

• sentence-transformers
• sentence-bert
• feature-extraction
• sentence-similarity

这是一个日语版Sentence-BERT模型。

此为日语专用Sentence-BERT模型（版本2）。

相较于版本1，本版本采用更优的损失函数MultipleNegativesRankingLoss进行训练优化。
在私有数据集测试中，本版本比版本1的准确率提升了1.5至2个百分点。

模型基于cl-tohoku/bert-base-japanese-whole-word-masking预训练模型构建。
运行推理需安装fugashi和ipadic（pip install fugashi ipadic）。

旧版本说明

https://qiita.com/sonoisa/items/1df94d0a98cd4f209051

只需将模型名称替换为"sonoisa/sentence-bert-base-ja-mean-tokens-v2"，即可使用本模型进行推理。

使用方法

from transformers import BertJapaneseTokenizer, BertModel  
import torch  

class SentenceBertJapanese:  
    def __init__(self, model_name_or_path, device=None):  
        self.tokenizer = BertJapaneseTokenizer.from_pretrained(model_name_or_path)  
        self.model = BertModel.from_pretrained(model_name_or_path)  
        self.model.eval()  

        if device is None:  
            device = "cuda" if torch.cuda.is_available() else "cpu"  
        self.device = torch.device(device)  
        self.model.to(device)  

    def _mean_pooling(self, model_output, attention_mask):  
        token_embeddings = model_output[0] #模型输出的第一个元素包含所有token嵌入  
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()  
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)  

    @torch.no_grad()  
    def encode(self, sentences, batch_size=8):  
        all_embeddings = []  
        iterator = range(0, len(sentences), batch_size)  
        for batch_idx in iterator:  
            batch = sentences[batch_idx:batch_idx + batch_size]  

            encoded_input = self.tokenizer.batch_encode_plus(batch, padding="longest",   
                                           truncation=True, return_tensors="pt").to(self.device)  
            model_output = self.model(**encoded_input)  
            sentence_embeddings = self._mean_pooling(model_output, encoded_input["attention_mask"]).to('cpu')  

            all_embeddings.extend(sentence_embeddings)  

        return torch.stack(all_embeddings)  

MODEL_NAME = "sonoisa/sentence-bert-base-ja-mean-tokens-v2"  # <- 此为v2版本  
model = SentenceBertJapanese(MODEL_NAME)  

sentences = ["暴走的AI", "失控的人工智能"]  
sentence_embeddings = model.encode(sentences, batch_size=8)  

print("句子嵌入向量：", sentence_embeddings)