许可证: llama3
语言:

• 英语
  任务标签: 文本生成
  标签:
• 英伟达
• chatqa-1.5
• chatqa
• llama-3
• pytorch

模型详情

我们推出Llama3-ChatQA-1.5，该模型在对话式问答（QA）和检索增强生成（RAG）方面表现卓越。Llama3-ChatQA-1.5基于ChatQA论文改进的训练方法开发，并构建于Llama-3基础模型之上。我们特别增加了更多对话式QA数据以增强其表格和算术计算能力。Llama3-ChatQA-1.5有两个版本：Llama3-ChatQA-1.5-8B和Llama3-ChatQA-1.5-70B。两个模型最初均使用Megatron-LM训练，后转换为Hugging Face格式。更多关于ChatQA的信息，请访问官网！

其他资源

Llama3-ChatQA-1.5-8B   评估数据   训练数据   检索器   官网   论文

基准测试结果

ChatRAG Bench中的结果如下：

注意：ChatQA-1.5基于Llama-3基础模型构建，而ChatQA-1.0基于Llama-2基础模型。ChatQA-1.5模型使用了HybriDial训练数据集。为确保公平比较，我们也比较了排除HybriDial后的平均分数。ChatRAG Bench的数据和评估脚本可在此处找到。

提示格式

强烈建议使用我们提供的提示格式，如下所示：

当上下文可用时

  
系统: {系统}  

{上下文}  

用户: {问题}  

助手: {回答}  

用户: {问题}  

助手:  

当上下文不可用时

  
系统: {系统}  

用户: {问题}  

助手: {回答}  

用户: {问题}  

助手:  

两种场景下系统回合的内容（即{系统}）如下：

  
这是用户与人工智能助手之间的对话。助手根据上下文提供有用、详细且有礼貌的回答。如果答案无法在上下文中找到，助手也应明确指出。  

请注意，我们的ChatQA-1.5模型针对带上下文的场景（如文档或检索到的上下文）进行了优化。

使用方法

将整个文档作为上下文

适用于整个文档可以放入模型的场景，无需对文档进行检索。

from transformers import AutoTokenizer, AutoModelForCausalLM  
import torch  

model_id = "nvidia/Llama3-ChatQA-1.5-70B"  

tokenizer = AutoTokenizer.from_pretrained(model_id)  
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")  

messages = [  
    {"role": "user", "content": "what is the percentage change of the net income from Q4 FY23 to Q4 FY24?"}  
]  

document = """NVIDIA (NASDAQ: NVDA) today reported revenue for the fourth quarter ended January 28, 2024, of $22.1 billion, up 22% from the previous quarter and up 265% from a year ago.\nFor the quarter, GAAP earnings per diluted share was $4.93, up 33% from the previous quarter and up 765% from a year ago. Non-GAAP earnings per diluted share was $5.16, up 28% from the previous quarter and up 486% from a year ago.\nQ4 Fiscal 2024 Summary\nGAAP\n| $ in millions, except earnings per share | Q4 FY24 | Q3 FY24 | Q4 FY23 | Q/Q | Y/Y |\n| Revenue | $22,103 | $18,120 | $6,051 | Up 22% | Up 265% |\n| Gross margin | 76.0% | 74.0% | 63.3% | Up 2.0 pts | Up 12.7 pts |\n| Operating expenses | $3,176 | $2,983 | $2,576 | Up 6% | Up 23% |\n| Operating income | $13,615 | $10,417 | $1,257 | Up 31% | Up 983% |\n| Net income | $12,285 | $9,243 | $1,414 | Up 33% | Up 769% |\n| Diluted earnings per share | $4.93 | $3.71 | $0.57 | Up 33% | Up 765% |"""  

def get_formatted_input(messages, context):  
    system = "System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context."  
    instruction = "Please give a full and complete answer for the question."  

    for item in messages:  
        if item['role'] == "user":  
            ## only apply this instruction for the first user turn  
            item['content'] = instruction + " " + item['content']  
            break  

    conversation = '\n\n'.join(["User: " + item["content"] if item["role"] == "user" else "Assistant: " + item["content"] for item in messages]) + "\n\nAssistant:"  
    formatted_input = system + "\n\n" + context + "\n\n" + conversation  

    return formatted_input  

formatted_input = get_formatted_input(messages, document)  
tokenized_prompt = tokenizer(tokenizer.bos_token + formatted_input, return_tensors="pt").to(model.device)  

terminators = [  
    tokenizer.eos_token_id,  
    tokenizer.convert_tokens_to_ids("<|eot_id|>")  
]  

outputs = model.generate(input_ids=tokenized_prompt.input_ids, attention_mask=tokenized_prompt.attention_mask, max_new_tokens=128, eos_token_id=terminators)  

response = outputs[0][tokenized_prompt.input_ids.shape[-1]:]  
print(tokenizer.decode(response, skip_special_tokens=True))  

运行检索以获取前n个片段作为上下文

适用于文档非常长的场景，此时需要进行检索。我们使用Dragon-multiturn检索器处理对话式查询。此外，我们还提供了一些文档供用户使用。

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModel  
import torch  
import json  

## 加载ChatQA-1.5分词器和模型  
model_id = "nvidia/Llama3-ChatQA-1.5-70B"  
tokenizer = AutoTokenizer.from_pretrained(model_id)  
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")  

## 加载检索器分词器和模型  
retriever_tokenizer = AutoTokenizer.from_pretrained('nvidia/dragon-multiturn-query-encoder')  
query_encoder = AutoModel.from_pretrained('nvidia/dragon-multiturn-query-encoder')  
context_encoder = AutoModel.from_pretrained('nvidia/dragon-multiturn-context-encoder')  

## 准备文档，以我们提供的路虎汽车手册文档为例  
chunk_list = json.load(open("docs.json"))['landrover']  

messages = [  
    {"role": "user", "content": "how to connect the bluetooth in the car?"}  
]  

### 运行检索  
## 将查询转换为以下格式：  
## user: {user}\nagent: {agent}\nuser: {user}  
formatted_query_for_retriever = '\n'.join([turn['role'] + ": " + turn['content'] for turn in messages]).strip()  

query_input = retriever_tokenizer(formatted_query_for_retriever, return_tensors='pt')  
ctx_input = retriever_tokenizer(chunk_list, padding=True, truncation=True, max_length=512, return_tensors='pt')  
query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :]  
ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :]  

## 使用点积计算相似度分数并排序  
similarities = query_emb.matmul(ctx_emb.transpose(0, 1)) # (1, num_ctx)  
ranked_results = torch.argsort(similarities, dim=-1, descending=True) # (1, num_ctx)  

## 获取前n个片段（n=5）  
retrieved_chunks = [chunk_list[idx] for idx in ranked_results.tolist()[0][:5]]  
context = "\n\n".join(retrieved_chunks)  

### 运行文本生成  
formatted_input = get_formatted_input(messages, context)  
tokenized_prompt = tokenizer(tokenizer.bos_token + formatted_input, return_tensors="pt").to(model.device)  

terminators = [  
    tokenizer.eos_token_id,  
    tokenizer.convert_tokens_to_ids("<|eot_id|>")