[Paper] ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

ScaleCUA represents a significant advancement in the field of computer use agents (CUAs) – AI systems designed to interact with graphical user interfaces (GUIs) just as humans do. Unlike previous approaches that often relied on proprietary models or limited datasets, ScaleCUA tackles one of the fundamental challenges in this domain: the scarcity of high-quality training data for teaching AI to navigate diverse computer interfaces.

The core innovation of ScaleCUA lies in its comprehensive approach to data collection and model training across multiple operating systems. The researchers have created a systematic pipeline that combines both automated agents and human experts to gather diverse interaction data from Windows, macOS, Linux, Android, iOS, and web platforms. This cross-platform approach enables the resulting models to generalize across different interfaces, making them significantly more versatile than previous systems limited to specific environments.

What makes this work particularly valuable to the AI community is its open-source nature. The authors have released not only their models but also their extensive dataset and training methodology, allowing other researchers to build upon their work and advance the field of computer automation further.

Summary of the Research

The ScaleCUA project addresses a critical bottleneck in developing effective computer use agents: the limited availability of GUI-centric training data. While image-text pairs are abundantly available on the internet for general vision-language models, computer interaction data is scarce, expensive to collect, and quickly becomes outdated as software interfaces evolve.

The researchers developed a novel “Cross-Platform Interactive Data Pipeline” with two synergistic components. First, an “Agent-Environment Interaction Loop” enables automated agents to interact with diverse GUI environments across six major platforms. Second, an “Agent-Human Hybrid Data Acquisition Loop” integrates expert-annotated trajectories to ensure comprehensive coverage and quality. This dual-loop approach produces rich screen-state observations, metadata (including accessibility trees and DOM structures), and interaction trajectories.

Building on this data collection framework, the team created a comprehensive training dataset covering three major task families:

1. GUI Understanding (471K examples) – Teaching models to comprehend interface elements, layouts, and text
2. GUI Grounding (17.1M examples) – Enabling accurate localization of UI elements based on descriptions
3. Task Completion (19K trajectories) – Training agents to perform multi-step interactions to achieve goals

Using this dataset, the researchers trained a series of models called ScaleCUA based on the Qwen2.5-VL architecture. These models support three distinct inference paradigms:

1. Grounding Mode – Focusing on precisely locating UI elements
2. Direct Action Mode – Generating executable actions without intermediate reasoning
3. Reasoned Action Mode – Producing a thought process before generating actions

The results demonstrate substantial improvements over previous state-of-the-art systems. ScaleCUA achieves 94.4% accuracy on MMBench-GUI L1-Hard (a GUI understanding benchmark), 60.6% on OSWorld-G (a grounding benchmark), and 47.4% on WebArena-Lite-v2 (an end-to-end task completion benchmark) – outperforming previous models by significant margins.

Key Findings and Contributions

The research presents several important findings that advance our understanding of computer use agents:

1. Cross-Platform Data is Crucial for General-Purpose CUAs

The study demonstrates that training on diverse interfaces across multiple operating systems significantly enhances generalization. Models trained on ScaleCUA’s cross-platform dataset showed consistent performance improvements across all benchmarks, confirming that exposure to varied GUI patterns enables better transfer learning.

2. Unified Action Space Enables Seamless Cross-Platform Operation

The researchers designed a standardized action space covering core operations (click, write, etc.) while accommodating platform-specific actions. This unified approach allows the model to generalize core behaviors while retaining the flexibility to exploit environment-specific actions, significantly simplifying downstream policy learning.

3. Data Augmentation and Weak Semantic Trajectories Improve Performance

Simple data augmentations like element cropping and screenshot concatenation increased performance by approximately 3.5% on grounding benchmarks. Additionally, incorporating “weak semantic trajectories” (interaction sequences without explicit high-level goals) provided valuable low-cost supervision for learning navigation patterns.

4. Explicit Reasoning Enhances Task Completion

The study found that models operating in “Reasoned Action Mode” (generating explicit reasoning steps before actions) consistently outperformed “Direct Action Mode” across all benchmarks. This suggests that incorporating structured thought processes helps maintain task coherence and reduce error propagation in long-horizon interactions.

5. Trade-offs Between General and GUI-specific Performance

The researchers observed that increasing the proportion of general-purpose multimodal data improved performance on general vision-language benchmarks but reduced GUI-specific accuracy. This highlights the need for carefully balanced training strategies to preserve GUI specialization while maintaining general reasoning capabilities.

These findings collectively provide a roadmap for developing more capable computer use agents, emphasizing the importance of diverse data, explicit reasoning, and balanced training approaches. The open-source release of ScaleCUA’s models, datasets, and training code represents a significant contribution to the research community, enabling further exploration and improvement of this technology.

The work demonstrates that scaling up training data in a targeted, domain-specific manner can yield substantial gains in AI capabilities, even when using relatively modest model architectures. This approach offers a promising direction for developing AI systems that can interact with digital interfaces as effectively as humans do, as explored in books like The AI Agent Blueprint and Transformers for Natural Language Processing and Computer Vision.