许可证: mit 语言:

• 英文 基础模型:
• Qwen/Qwen2.5-VL-3B-Instruct 管道标签: 视觉问答

简介

本仓库包含高效GUI定位模型UI-R1-E-3B，出自论文UI-R1: 通过强化学习增强GUI代理的行为预测。

项目主页: https://github.com/lll6gg/UI-R1

旧版本: UI-R1-3B

基准测试1: ScreenSpotV2

基准测试2: ScreenSpot-Pro

排行榜: UI-I2E-Bench

GUI定位评估代码

1. UI-R1-E-3B生成代码：

   model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
       args.model_path,
       torch_dtype=torch.bfloat16,
       attn_implementation="flash_attention_2",
       device_map="cpu",
   )
   model = model.to(torch.device(rank))
   model = model.eval()
   processor = AutoProcessor.from_pretrained(ori_processor_path)
   question_template = (
       f"在此UI截图中，我想执行命令'{task_prompt}'。\n"
       "请提供要执行的操作(枚举['click'])"
       "以及如果执行点击时光标移动到的坐标(整数)。\n"
       "直接在<answer></answer>标签中输出最终答案。"
       "输出答案格式应如下：\n"
       "<answer>[{'action': 'click', 'coordinate': [x, y]}]</answer>\n"
       "请严格遵守格式。"
   )
   query = '<image>\n' + question_template
   messages = [
       {
           "role": "user",
           "content": [
               {"type": "image", "image": image_path}
           ] + [{"type": "text", "text": query}],
       }
   ]
   text = processor.apply_chat_template(
       messages, tokenize=False, add_generation_prompt=True
   )
   image_inputs, video_inputs = process_vision_info(messages)
   inputs = processor(
       text=[text],
       images=image_inputs,
       videos=video_inputs,
       padding=True,
       return_tensors="pt",
   )
   generated_ids = model.generate(**inputs, max_new_tokens=1024)
   generated_ids_trimmed = [
       out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
   ]
   response = processor.batch_decode(
       generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
   )
   response = response[0]
   pred_coord, _ = extract_coord(response)
   

2. 根据图像缩放比例调整预测坐标

   image = Image.open(image_path)
   origin_width, origin_height = image.size
   resized_height,resized_width = smart_resize(origin_height,origin_width,max_pixels=12845056)
   scale_x = origin_width / resized_width
   scale_y = origin_height / resized_height
   pred_coord[0] = int(pred_coord[0] * scale_x)
   pred_coord[1] = int(pred_coord[1] * scale_y)
   

   智能缩放函数来自Qwen2VL：

   import math
   def smart_resize(
       height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
   ):
       """缩放图像以满足以下条件：
   
       1. 高度和宽度都能被'factor'整除
   
       2. 总像素数在['min_pixels', 'max_pixels']范围内
   
       3. 尽可能保持图像宽高比
       """
       if height < factor or width < factor:
           raise ValueError(f"高度:{height}或宽度:{width}必须大于因子:{factor}")
       elif max(height, width) / min(height, width) > 200:
           raise ValueError(
               f"绝对宽高比必须小于200，当前为{max(height, width) / min(height, width)}"
           )
       h_bar = round(height / factor) * factor
       w_bar = round(width / factor) * factor
       if h_bar * w_bar > max_pixels:
           beta = math.sqrt((height * width) / max_pixels)
           h_bar = math.floor(height / beta / factor) * factor
           w_bar = math.floor(width / beta / factor) * factor
       elif h_bar * w_bar < min_pixels:
           beta = math.sqrt(min_pixels / (height * width))
           h_bar = math.ceil(height * beta / factor) * factor
           w_bar = math.ceil(width * beta / factor) * factor
       return h_bar, w_bar