基础模型:

• TareksLab/L2-MERGE2a
• TareksLab/L2-MERGE4
• TareksLab/L-BASE-V1
• TareksLab/L2-MERGE3
• TareksLab/L2-MERGE1 库名称: transformers 标签:
• 合并工具包
• 模型合并

模型合并

这是使用mergekit工具对预训练语言模型进行合并的结果。

合并详情

合并方法

本模型采用DARE TIES合并方法，以TareksLab/L-BASE-V1作为基础模型进行融合。

参与合并的模型

合并包含以下模型：

• TareksLab/L2-MERGE2a
• TareksLab/L2-MERGE4
• TareksLab/L2-MERGE3
• TareksLab/L2-MERGE1

配置参数

生成本模型所使用的YAML配置如下：

models:
  - model: TareksLab/L2-MERGE4
    parameters:
      weight: 
      - filter: self_attn
        value: [0.3, 0.1, 0.2]
      - filter: mlp
        value: [0.4, 0.2, 0.1]
      - value: 0.2
      density: 0.7
      lambda: 1.05
  - model: TareksLab/L2-MERGE2a
    parameters:
      weight: 
      - filter: self_attn
        value: [0.2, 0.1, 0.3]
      - filter: mlp
        value: [0.3, 0.1, 0.2]
      - value: 0.2
      density: 0.65
      lambda: 1.05
  - model: TareksLab/L2-MERGE3
    parameters:
      weight: 
      - filter: self_attn
        value: [0.1, 0.3, 0.1]
      - filter: mlp
        value: [0.2, 0.3, 0.1]
      - value: 0.2
      density: 0.6
      lambda: 1.05
  - model: TareksLab/L2-MERGE1
    parameters:
      weight: 
      - filter: self_attn
        value: [0.2, 0.2, 0.1]
      - filter: mlp
        value: [0.1, 0.2, 0.2]
      - value: 0.2
      density: 0.6
      lambda: 1
  - model: TareksLab/L-BASE-V1
    parameters:
      weight: 
      - filter: self_attn
        value: [0.1, 0.3, 0.3]
      - filter: mlp
        value: [0.1, 0.2, 0.4]
      - value: 0.2
      density: 0.55
      lambda: 1
base_model: TareksLab/L-BASE-V1
merge_method: dare_ties
parameters:
  normalize: false
  pad_to_multiple_of: 4
tokenizer:
  source: base
chat_template: llama3
dtype: bfloat16