Usage

Python library: parse a model response into pyautogui code

This is the minimal end-to-end flow using the ui-tars Python package. You provide the raw text output from the VLM inference call; the library parses it into a structured dict and then into runnable pyautogui code.

from ui_tars.action_parser import (
    parse_action_to_structure_output,
    parsing_response_to_pyautogui_code,
)

# A raw response string from the UI-TARS VLM
response = "Thought: Click the submit button\nAction: click(start_box='(100,200)')"

original_image_width = 1920
original_image_height = 1080

# Step 1: parse the response into a structured dict
parsed_dict = parse_action_to_structure_output(
    response,
    factor=1000,
    origin_resized_height=original_image_height,
    origin_resized_width=original_image_width,
    model_type="qwen25vl",
)
print(parsed_dict)

# Step 2: convert the structured dict into executable pyautogui code
parsed_code = parsing_response_to_pyautogui_code(
    responses=parsed_dict,
    image_height=original_image_height,
    image_width=original_image_width,
)
print(parsed_code)

Notes:

• factor=1000 maps normalized coordinates (0-1000 scale) back to pixel space.
• model_type must match the VLM backend: "qwen25vl" for Qwen 2.5-VL based checkpoints (UI-TARS-1.5-*), "qwen2vl" for older checkpoints.
• The output of parsing_response_to_pyautogui_code is a Python code string ready to be exec()-ed or written to a script.

Prompt templates

Three built-in system-prompt templates live in ui_tars/prompt.py:

Template	Use case
COMPUTER_USE	Desktop GUI automation (Windows, Linux, macOS)
MOBILE_USE	Android emulator tasks
GROUNDING	Action-only output for evaluation benchmarks

Import and use them when constructing your VLM inference request:

from ui_tars.prompt import COMPUTER_USE

system_prompt = COMPUTER_USE

Agent TARS CLI: run a task with a cloud model

Connect to Anthropic Claude as the backend (requires an Anthropic API key):

agent-tars --provider anthropic --model claude-3-7-sonnet-latest --apiKey YOUR_API_KEY

Connect to Volcengine Doubao (ByteDance cloud, works well with UI-TARS weights):

agent-tars --provider volcengine --model doubao-1-5-thinking-vision-pro-250428 --apiKey YOUR_API_KEY

Desktop application

Launch the downloaded binary. On first run, configure the model endpoint (local vLLM or remote API) in the settings panel. Then type a natural-language task in the task input field and press Run.