基于字符级别并在enwik8上训练的Reformer语言模型

enwik8是一个基于维基百科的数据集，常用于衡量模型压缩数据的能力，例如在Hutter奖的范畴内：https://en.wikipedia.org/wiki/Hutter_Prize。

reformer-enwik8模型在enwik8前9000万个字符上进行了预训练，文本被分块为65536个字符（=2^16）的批次。模型权重取自https://console.cloud.google.com/storage/browser/trax-ml/reformer/enwik8，并转换为Hugging Face的PyTorch ReformerLM模型ReformerModelWithLMHead。

该模型是一个在字符级别上操作的语言模型。因此，该模型不需要分词器。可以使用以下函数进行编码和解码：

import torch

# 编码
def encode(list_of_strings, pad_token_id=0):
    max_length = max([len(string) for string in list_of_strings])

    # 创建空张量
    attention_masks = torch.zeros((len(list_of_strings), max_length), dtype=torch.long)
    input_ids = torch.full((len(list_of_strings), max_length), pad_token_id, dtype=torch.long)

    for idx, string in enumerate(list_of_strings):
        # 确保字符串为字节格式
        if not isinstance(string, bytes):
            string = str.encode(string)

        input_ids[idx, :len(string)] = torch.tensor([x + 2 for x in string])
        attention_masks[idx, :len(string)] = 1

    return input_ids, attention_masks
    
# 解码
def decode(outputs_ids):
    decoded_outputs = []
    for output_ids in outputs_ids.tolist():
        # 将ID转换回字符，ID小于2的转换为空字符串
        decoded_outputs.append("".join([chr(x - 2) if x > 1 else "" for x in output_ids]))
    return decoded_outputs

可以按以下方式生成文本：

from transformers import ReformerModelWithLMHead

model = ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8")
encoded, attention_masks = encode(["In 1965, Brooks left IBM to found the Department of"])
decode(model.generate(encoded, do_sample=True, max_length=150))

# 输出：
# In 1965, Brooks left IBM to found the Department of Journalism in 1968. IBM had jurisdiction himself in 1980, while Brooks resolved, nevertheless thro

注意：使用ReformerModelWithLMHead进行语言生成尚未优化，速度较慢。