UI-TARS SOP

Appearance

Test ​

Status: PASS (action-parsing SDK) / NEEDS-ACCOUNT (model weights) ​

Two distinct components exist under the UI-TARS name. The ui-tars pip package is an action-parsing utility SDK that installs and runs without any model or GPU. The ByteDance UI-TARS model weights (7B/72B) are separate and require HuggingFace access and significant GPU resources.

Component 1: ui-tars pip package (PASS) ​

Install command ​

bash
pip install ui-tars

Import check ​

bash
python -c "import ui_tars; print('ui_tars imported OK')"

Actual output ​

ui_tars imported OK

Package details ​

  • Package: ui-tars
  • Version: 0.5.1
  • Authors: liangshihao.0828@bytedance.com (ByteDance)
  • Summary: "Parsing LLM-generated GUI action instructions, automatically generating pyautogui scripts, and supporting coordinate conversion and smart image resizing."

This is the action-parsing SDK, not the model weights. It parses LLM-generated action strings and converts them to pyautogui code.

Smoke test: action parser end-to-end ​

After pip install ui-tars, run this to confirm the parser works without a live model:

bash
python - <<'EOF'
from ui_tars.action_parser import (
    parse_action_to_structure_output,
    parsing_response_to_pyautogui_code,
)

response = "Thought: Click the button\nAction: click(start_box='(100,200)')"
parsed = parse_action_to_structure_output(
    response,
    factor=1000,
    origin_resized_height=1080,
    origin_resized_width=1920,
    model_type="qwen25vl",
)
code = parsing_response_to_pyautogui_code(
    responses=parsed,
    image_height=1080,
    image_width=1920,
)
print("PARSED:", parsed)
print("CODE:", code)
print("OK")
EOF

Expected success output:

PARSED: [{'action_type': 'click', 'action_inputs': {'start_box': '(100,200)', ...}}]
CODE: import pyautogui
pyautogui.click(x=192, y=216)
OK

The exact pixel coordinates will vary based on the factor and resolution math. Key signals: no ImportError, a PARSED: line with action_type and action_inputs keys, a CODE: line with a valid pyautogui.click(x=..., y=...) call, and the final OK line.

Component 2: ByteDance UI-TARS model weights (NEEDS-ACCOUNT) ​

The model itself (7B/72B checkpoints) requires:

  1. A GPU instance with at least 24 GB VRAM for the 7B model
  2. HuggingFace account and huggingface-cli login to download the bytedance-research/UI-TARS-1.5-7B checkpoint
  3. vLLM or HF Inference Endpoint with the checkpoint loaded
  4. A screen capture pipeline (pyautogui + PIL on a desktop OS)

These are not runnable in a standard CI or sandbox environment.

Agent TARS CLI (optional, separate project) ​

bash
npx @agent-tars/cli@latest --version

Expected success output:

@agent-tars/cli/x.y.z ...

Any semver string printed without error confirms the CLI is available.

Classification ​

PASS for the ui-tars pip package (action-parsing SDK, installable, no GPU required). NEEDS-ACCOUNT for the UI-TARS model weights (HuggingFace download required, GPU needed for inference).