pipeline_tag: 句子相似度 language: 英文 license: apache-2.0 tags:

• 句子转换器
• 特征提取
• 句子相似度
• 转换器

hku-nlp/instructor-base

这是一个通用嵌入模型：它能在测试阶段无需额外训练，将任意文本片段（如标题、句子、文档等）映射为固定长度的向量。通过指令引导，生成的嵌入向量具有领域特异性（例如专为科学、金融等领域优化）和任务感知性（例如针对分类、信息检索等任务定制）。

该模型可轻松通过sentence-transformer库调用。

安装步骤

git clone https://github.com/HKUNLP/instructor-embedding
cd sentence-transformers
pip install -e .

生成定制化嵌入向量

随后可按如下方式计算领域特定且任务感知的嵌入向量：

from sentence_transformers import SentenceTransformer
sentence = "3D ActionSLAM: wearable person tracking in multi-floor environments"
instruction = "Represent the Science title; Input:"
model = SentenceTransformer('hku-nlp/instructor-base')
embeddings = model.encode([[instruction,sentence,0]])
print(embeddings)

计算句子相似度

您还可以进一步使用该模型计算两组句子间的相似度，并生成定制化嵌入向量。

from sklearn.metrics.pairwise import cosine_similarity
sentences_a = [['Represent the Science sentence; Input: ','Parton energy loss in QCD matter',0], 
               ['Represent the Financial statement; Input: ','The Federal Reserve on Wednesday raised its benchmark interest rate.',0]
sentences_b = [['Represent the Science sentence; Input: ','The Chiral Phase Transition in Dissipative Dynamics', 0],
               ['Represent the Financial statement; Input: ','The funds rose less than 0.5 per cent on Friday',0]
embeddings_a = model.encode(sentences_a)
embeddings_b = model.encode(sentences_b)
similarities = cosine_similarity(embeddings_a,embeddings_b)
print(similarities)