language:

• ko pipeline_tag: text2text-generation license: apache-2.0

韩语拼写校正器(Korean Typos Corrector)

• 基于ETRI-et5模型微调的韩语口语专用拼写校正工具

基于PLM模型(ET5)

• ETRI(https://aiopen.etri.re.kr/et5Model)

基于数据集

• 国民语料库(https://corpus.korean.go.kr/request/reausetMain.do?lang=ko)拼写校正数据

数据预处理

• 1. 去除特殊符号（逗号）和句号
• 2. 清除空值("")
• 3. 过滤过短句子（长度≤2字符）
• 4. 删除包含&name&、name1等姓名标签的词汇（仅删除标签保留原句）
• 总计：318,882组数据对

使用方法

from transformers import T5ForConditionalGeneration, T5Tokenizer

# 加载T5模型
model = T5ForConditionalGeneration.from_pretrained("j5ng/et5-typos-corrector")
tokenizer = T5Tokenizer.from_pretrained("j5ng/et5-typos-corrector")

device = "cuda:0" if torch.cuda.is_available() else "cpu"
# device = "mps:0" if torch.cuda.is_available() else "cpu" # 适用于mac m1芯片

model = model.to(device) 

# 示例输入
input_text = "아늬 진짜 무ㅓ하냐고"

# 编码输入文本
input_encoding = tokenizer("请修正拼写: " + input_text, return_tensors="pt")

input_ids = input_encoding.input_ids.to(device)
attention_mask = input_encoding.attention_mask.to(device)

# 生成修正结果
output_encoding = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_length=128,
    num_beams=5,
    early_stopping=True,
)

# 解码输出文本
output_text = tokenizer.decode(output_encoding[0], skip_special_tokens=True)

# 打印结果
print(output_text) # 输出：아니 진짜 뭐 하냐고.

使用Transformers流水线

from transformers import T5ForConditionalGeneration, T5Tokenizer, pipeline

model = T5ForConditionalGeneration.from_pretrained('j5ng/et5-typos-corrector')
tokenizer = T5Tokenizer.from_pretrained('j5ng/et5-typos-corrector')

typos_corrector = pipeline(
    "text2text-generation",
    model=model,
    tokenizer=tokenizer,
    device=0 if torch.cuda.is_available() else -1,
    framework="pt",
)

input_text = "완죤 어이업ㅅ네진쨬ㅋㅋㅋ"
output_text = typos_corrector("请修正拼写: " + input_text,
            max_length=128,
            num_beams=5,
            early_stopping=True)[0]['generated_text']

print(output_text) # 输出：완전 어이없네 진짜 ㅋㅋㅋ.