金融RAG套娃模型

基于Alibaba-NLP/gte-large-en-v1.5微调的金融专用模型。该模型能将句子和段落映射到1024维密集向量空间，可用于语义文本相似度、语义搜索、复述挖掘、文本分类、聚类等任务。

本模型在保持通用性能的同时，致力于在金融文档检索任务中表现出色。

模型详情

模型描述

• 模型类型: 句子转换器
• 基础模型: Alibaba-NLP/gte-large-en-v1.5
• 最大序列长度: 8192个token
• 输出维度: 1024个token
• 相似度函数: 余弦相似度
• 语言: 英语
• 许可证: Apache-2.0

模型来源

• 文档: Sentence Transformers文档
• 代码库: GitHub上的Sentence Transformers
• Hugging Face: Hugging Face上的Sentence Transformers

完整模型架构

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

使用方法

直接使用(Sentence Transformers)

首先安装Sentence Transformers库:

pip install -U sentence-transformers

然后加载模型并运行推理:

from sentence_transformers import SentenceTransformer

# 从Hugging Face Hub下载
model = SentenceTransformer("rbhatia46/gte-large-en-v1.5-financial-rag-matryoshka")
# 运行推理
sentences = [
    'JP Morgan reported total deposits of $2.6 trillion in the year ending December 31, 2023.',
    "What were JP Morgan's total deposits in 2023?",
    'What is the primary source of revenue for the software company, Microsoft?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 获取嵌入向量的相似度分数
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

评估

评估指标

信息检索(1024维)

(其他维度的评估指标表格类似，此处省略)

训练详情

训练数据集

• 大小: 4,275个训练样本
• 列: positive和anchor
• 损失函数: 使用MatryoshkaLoss配合MultipleNegativesRankingLoss

训练超参数

• 评估策略: epoch
• 每设备训练批量大小: 32
• 每设备评估批量大小: 16
• 梯度累积步数: 16
• 学习率: 2e-05
• 训练周期数: 10
• 学习率调度器类型: cosine
• 预热比例: 0.1
• 使用bf16: True
• 使用tf32: True
• 结束时加载最佳模型: True
• 优化器: adamw_torch_fused
• 批量采样器: no_duplicates

引用

BibTeX引用

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}