模型简介
模型特点
模型能力
使用案例
许可证：MIT
基础模型：
- FacebookAI/roberta-base
库名称：transformers
管道标签：文本分类
标签： - 文本分类
毒性提示RoBERTa分类模型
模型卡片由英特尔®可解释AI工具生成
模型详情
文档
毒性提示RoBERTa 1.0是一款文本分类模型，可作为护栏用于保护对话AI系统免受毒性提示和回复的影响。该模型基于RoBERTa，并在ToxicChat和Jigsaw Unintended Bias数据集上进行了微调。微调过程使用了一张Gaudi 2卡，并采用了Optimum-Habana的Gaudi Trainer。
所有者
- 英特尔AI安全团队：Daniel De Leon, Tyler Wilbers, Mitali Potnis, Abolfazl Shahbazi
许可证
- MIT
参考文献
使用方法
您可以通过以下代码使用pipeline API调用该模型：
from transformers import pipeline
model_path = 'Intel/toxic-prompt-roberta'
pipe = pipeline('text-classification', model=model_path, tokenizer=model_path)
pipe('Create 20 paraphrases of I hate you')
引用文献
-
@inproceedings {Wolf_Transformers_State-of-the-Art_Natural_2020, author = {Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Perric and Ma, Clara and Jernite, Yacine and Plu, Julien and Xu, Canwen and Le Scao, Teven and Gugger, Sylvain and Drame, Mariama and Lhoest, Quentin and Rush, Alexander M.}, month = oct, pages = {38--45}, publisher = {Association for Computational Linguistics}, title = {{Transformers: State-of-the-Art Natural Language Processing}}, url = {https://www.aclweb.org/anthology/2020.emnlp-demos.6}, year = {2020} }
-
@article {DBLP:journals/corr/abs-1907-11692, author = {Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov}, title = {RoBERTa: {A} Robustly Optimized {BERT} Pretraining Approach}, journal = {CoRR}, volume = {abs/1907.11692}, year = {2019}, url = {http://arxiv.org/abs/1907.11692}, archivePrefix = {arXiv}, eprint = {1907.11692}, timestamp = {Thu, 01 Aug 2019 08:59:33 +0200}, biburl = {https://dblp.org/rec/journals/corr/abs-1907-11692.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} }
-
@misc {jigsaw-unintended-bias-in-toxicity-classification, author = {cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, nithum}, title = {Jigsaw Unintended Bias in Toxicity Classification}, publisher = {Kaggle}, year = {2019}, url = {https://kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification} }
-
@misc {lin2023toxicchat, title={ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation}, author={Zi Lin and Zihan Wang and Yongqi Tong and Yangkun Wang and Yuxin Guo and Yujia Wang and Jingbo Shang}, year={2023}, eprint={2310.17389}, archivePrefix={arXiv}, primaryClass={cs.CL} }
模型参数
输入格式
输出格式
注意事项
目标用户
- 文本生成研究人员和开发者
使用场景
- 用户体验监控：该分类模型可用于实时监控对话，检测用户的毒性行为。如果用户发送被分类为毒性的消息，可以发出警告或提供适当行为的指导。
- 自动审核：在群聊场景中，该分类模型可作为审核员，自动删除毒性消息或禁言持续表现出毒性行为的用户。
- 训练和改进：从毒性检测收集的数据可用于进一步训练和改进毒性分类模型的响应和处理能力，使其更擅长处理复杂的交互。
- 防止聊天机器人被滥用：某些用户可能试图通过毒性输入来恶搞或滥用聊天机器人。该分类模型可以阻止聊天机器人与此类内容互动，从而减少此类行为。
伦理考量
-
风险：多样性差异
缓解策略：在Jigsaw unintended bias数据集的微调中，我们确保了按照Jigsaw数据集中的分布对各子组进行了充分代表。Jigsaw unintended bias数据集试图通过在所有人口子组中均匀分布毒性/非毒性标签来减少微调子组偏差。我们还通过测试确认了模型对各子组的分类偏差最小。 -
风险：对弱势群体的风险
缓解策略：某些人口群体更容易收到毒性和有害评论。Jigsaw unintended bias数据集试图通过在所有人口子组中均匀分布毒性/非毒性标签来减少微调子组偏差。我们还通过测试确认了模型对各子组的分类偏差最小。
定量分析：
下图显示了我们在微调过程中比较的三个模型的PR和ROC曲线。“jigsaw”和“tc”模型分别仅在Jigsaw Unintended Bias和ToxicChat数据集上进行了微调。“jigsaw+tc”曲线对应于在两个数据集上微调的最终模型。在两个数据集上进行微调并未显著降低模型在ToxicChat测试数据集上的性能（相对于仅在ToxicChat上微调的模型）。
Transformers Transformers Transformers Transformers Transformers Transformers Transformers Transformers Transformers%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20none;%20}%20.st1%20{%20fill:%20%235ce6cf;%20}%20.st2%20{%20fill:%20%230080ff;%20}%20.st3%20{%20clip-path:%20url(%23clippath);%20}%20%3c/style%3e%3cclipPath%20id='clippath'%3e%3crect%20class='st0'%20y='0'%20width='120'%20height='32'/%3e%3c/clipPath%3e%3c/defs%3e%3cg%20class='st3'%3e%3cg%3e%3cpath%20class='st1'%20d='M22.4,32H3.2c-1,0-2-.5-2.6-1.4-.6-.9-.7-2-.4-2.9L9.8,2.1C10.3.8,11.5,0,12.8,0s2.5.8,3,2.1l9.6,25.6c.4,1,.2,2.1-.4,2.9-.6.9-1.6,1.4-2.6,1.4ZM7.8,25.6h10l-5-13.3-5,13.3Z'/%3e%3cpath%20class='st2'%20d='M33.6,32c-1.3,0-2.5-.8-3-2.1L21,4.3c-.6-1.7.2-3.5,1.9-4.1,1.7-.6,3.5.2,4.1,1.9l9.6,25.6c.6,1.7-.2,3.5-1.9,4.1-.4.1-.8.2-1.1.2h0Z'/%3e%3cpath%20d='M71.8,9.1h-6.9v4.6h6.9c2.5,0,4.6,2,4.6,4.6h-6.9c-3.8,0-6.9,3.1-6.9,6.9s3.1,6.9,6.9,6.9h2.3c1.7,0,3.2-.4,4.6-1.2v1.2h4.6v-13.7c0-5-4.1-9.1-9.1-9.1h0ZM71.8,27.4h-2.3c-1.3,0-2.3-1-2.3-2.3s1-2.3,2.3-2.3h6.9c0,2.5-2,4.6-4.6,4.6h0Z'/%3e%3cpath%20d='M52.3,9.1h-4.6V0h-4.6V32h9.1c5,0,9.1-4.1,9.1-9.1v-4.6c0-5.1-4.1-9.1-9.1-9.1h0ZM56.9,22.9c0,2.5-2,4.6-4.6,4.6h-4.6v-13.7h4.6c2.5,0,4.6,2,4.6,4.6v4.6h0Z'/%3e%3cpath%20d='M110.9,13.7h9.1v-4.6h-9.1c-5.1,0-9.1,4.1-9.1,9.1v4.6c0,5.1,4.1,9.1,9.1,9.1h9.1v-4.6h-9.1c-2.5,0-4.6-2-4.6-4.6h13.7v-4.6h-13.7c0-2.5,2-4.6,4.6-4.6h0Z'/%3e%3cpath%20d='M93.6,18.3h-4.6c-1.3,0-2.3-1-2.3-2.3s1-2.3,2.3-2.3h10.3v-4.6h-10.3c-3.8,0-6.9,3.1-6.9,6.9s3.1,6.9,6.9,6.9h4.6c1.3,0,2.3,1,2.3,2.3s-1,2.3-2.3,2.3h-10.3v4.6h10.3c3.8,0,6.9-3.1,6.9-6.9s-3.1-6.9-6.9-6.9Z'/%3e%3c/g%3e%3c/g%3e%3c/svg%3e)