🚀 BERT-tiny在GooAQ上训练的模型

这是一个基于Cross Encoder的模型，它使用sentence-transformers库从prajjwal1/bert-tiny微调而来。该模型可以为文本对计算得分，可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等任务。

此模型使用train_script.py进行训练。

🚀 快速开始

本模型是一个基于Cross Encoder的微调模型，可用于计算文本对的得分，适用于多种自然语言处理任务。下面将介绍如何安装依赖库并使用该模型进行推理。

✨ 主要特性

• 多任务适用性：可用于语义文本相似度、语义搜索、释义挖掘、文本分类、聚类等多种任务。
• 微调模型：基于prajjwal1/bert-tiny进行微调，能更好地适应特定任务。

📦 安装指南

首先，你需要安装Sentence Transformers库：

pip install -U sentence-transformers

💻 使用示例

基础用法

安装好库后，你可以加载模型并进行推理：

from sentence_transformers import CrossEncoder

# 从🤗 Hub下载模型
model = CrossEncoder("cross-encoder-testing/reranker-bert-tiny-gooaq-bce")
# 定义文本对
pairs = [
    ['are javascript developers in demand?', "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL."],
    ['are javascript developers in demand?', 'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.'],
    ['are javascript developers in demand?', 'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.'],
    ['are javascript developers in demand?', 'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.'],
    ['are javascript developers in demand?', 'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.'],
]
# 预测得分
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# 或者根据与单个文本的相似度对不同文本进行排序
ranks = model.rank(
    'are javascript developers in demand?',
    [
        "JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL.",
        'In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture.',
        'Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language.',
        'Things in the body tag are the things that should be displayed: the actual content. Javascript in the body is executed as it is read and as the page is rendered. Javascript in the head is interpreted before anything is rendered.',
        'Web apps tend to be built using JavaScript, CSS and HTML5. Unlike mobile apps, there is no standard software development kit for building web apps. However, developers do have access to templates. Compared to mobile apps, web apps are usually quicker and easier to build — but they are much simpler in terms of features.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

📚 详细文档

模型详情

模型资源

• 文档：Sentence Transformers Documentation
• 文档：Cross Encoder Documentation
• 仓库：Sentence Transformers on GitHub
• Hugging Face：Cross Encoders on Hugging Face

评估指标

Cross Encoder重排序

• 数据集：gooaq-dev、NanoMSMARCO、NanoNFCorpus和NanoNQ
• 评估方法：使用CrossEncoderRerankingEvaluator进行评估

Cross Encoder Nano BEIR

• 数据集：NanoBEIR_R100_mean
• 评估方法：使用CrossEncoderNanoBEIREvaluator进行评估

训练详情

训练数据集

• 未命名数据集
  • 大小：578,402个训练样本
  • 列：question、answer和label
  • 基于前1000个样本的近似统计信息： | | 问题 | 答案 | 标签 | | ---- | ---- | ---- | ---- | | 类型 | string | string | int | | 详情 |
    • 最小长度: 21个字符
    • 平均长度: 43.81个字符
    • 最大长度: 96个字符
    |
    • 最小长度: 51个字符
    • 平均长度: 252.46个字符
    • 最大长度: 405个字符
    |
    • 0: ~82.90%
    • 1: ~17.10%
    |
  • 样本： | 问题 | 答案 | 标签 | | ---- | ---- | ---- | | are javascript developers in demand? | JavaScript is the skill that is most in-demand for IT in 2020, according to a report from developer skills tester DevSkiller. The report, “Top IT Skills report 2020: Demand and Hiring Trends,” has JavaScript switching places with Java when compared to last year's report, with Java in third place this year, behind SQL. | 1 | | are javascript developers in demand? | In one line difference between the two is: JavaScript is the programming language where as AngularJS is a framework based on JavaScript. ... It is also the basic for all java script based technologies like jquery, angular JS, bootstrap JS and so on. Angular JS is a framework written in javascript and uses MVC architecture. | 0 | | are javascript developers in demand? | Java applications are run in a virtual machine or web browser while JavaScript is run on a web browser. Java code is compiled whereas while JavaScript code is in text and in a web page. JavaScript is an OOP scripting language, whereas Java is an OOP programming language. | 0 |
• 损失函数：使用BinaryCrossEntropyLoss，参数如下：

{
    "activation_fct": "torch.nn.modules.linear.Identity",
    "pos_weight": 5
}

训练超参数

非默认超参数

• eval_strategy: steps
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• learning_rate: 0.0005
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True

所有超参数

训练日志

环境影响

使用CodeCarbon测量碳排放：

• 能耗：0.019 kWh
• 碳排放：0.007 kg的CO2
• 使用时长：0.099小时

训练硬件

• 是否使用云服务：否
• GPU型号：1 x NVIDIA GeForce RTX 3090
• CPU型号：13th Gen Intel(R) Core(TM) i7-13700K
• 内存大小：31.78 GB

框架版本

• Python: 3.11.6
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.48.3
• PyTorch: 2.5.0+cu121
• Accelerate: 1.3.0
• Datasets: 2.20.0
• Tokenizers: 0.21.0

🔧 技术细节

本模型基于Cross Encoder架构，使用BinaryCrossEntropyLoss作为损失函数进行微调。通过对特定数据集的训练，模型能够学习到文本对之间的语义关系，从而为文本对计算得分。在训练过程中，使用了一系列超参数来控制训练过程，如学习率、批次大小等。

📄 许可证

本模型使用apache-2.0许可证。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}