标签:

• 羊驼模型
• sh2orc

基础模型:

• meta-llama/Meta-Llama-3.1-8B-Instruct

Llama-3.1-韩语-8B-指令版

Llama-3.1-韩语-8B-指令版是基于Meta-Llama-3.1微调的模型：

• meta-llama/Meta-Llama-3.1-8B-Instruct

• 训练数据集：
  • maywell/ko_wikidata_QA
  • lcw99/wikipedia-korean-20240501-1million-qna
  • jojo0217/korean_rlhf_dataset
  • MarkrAI/KoCommercial-Dataset

💻 Transformers使用指南

使用transformers库 从transformers >= 4.43.0版本开始，您可以使用Transformers的pipeline抽象或通过Auto类配合generate()函数进行对话推理。 请确保通过pip install --upgrade transformers更新您的transformers安装。

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "sh2orc/Llama-3.1-Korean-8B-Instruct"
messages = [{"role": "user", "content": "什么是大语言模型？"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=2048, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

💻 VLLM使用指南

使用vllm库 从vllm版本开始，您可以使用vLLM的pipeline抽象配合gen()函数进行对话推理。 请确保通过pip install --upgrade vllm更新您的vllm安装。

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer, pipeline

BASE_MODEL = "sh2orc/Llama-3.1-Korean-8B-Instruct"

llm = LLM(model=BASE_MODEL)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = 'right'

def gen(instruction):
    messages = [
        {
          "role": "system",
          "content": "您是一位出色的AI助手。如果不知道用户问题的答案，请直接告知不了解。You are a great AI assistant. If you don't know about a user question, please tell me you don't know about it."
        },
        {
          "role": "user",
          "content": instruction
        }, 
    ]
    
    prompt_message = tokenizer.apply_chat_template(
            messages, 
            tokenize=False, 
            add_generation_prompt=True,
    )
    
    eos_token_id = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|eot_id|>")]
    
    outputs = llm.generate(prompt_message, SamplingParams(
        stop_token_ids=eos_token_id, 
        temperature=0.3, 
        top_p=0.9,
        frequency_penalty=1.3,
        max_tokens=4096))
    
    for output in outputs:
        propt = output.prompt
        generated_text = output.outputs[0].text
        print(generated_text)

问答示例