[Paper] The Unreasonable Effectiveness of Scaling Agents for Computer Use

Computer-use agents (CUAs) have emerged as a promising frontier in automating everyday digital tasks, yet their broader application has been limited by unreliability and high variance across complex, long-horizon tasks. A groundbreaking research paper from Simular Research titled “The Unreasonable Effectiveness of Scaling Agents for Computer Use” introduces a novel framework called Behavior Best-of-N (bBoN) that achieves near-human performance on computer tasks through intelligent scaling of agent capabilities. The researchers from Simular—Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang—demonstrate that by generating multiple solution attempts and intelligently selecting among them, their approach substantially improves robustness and success rates, reaching 69.9% accuracy on benchmark tasks and approaching human-level performance at 72%.

The core insight of the paper revolves around the concept of “wide scaling” for computer-use agents. Traditional approaches to improving AI agent capabilities have focused on making individual agents smarter or more sophisticated. In contrast, bBoN takes a fundamentally different approach by generating multiple independent solution attempts (rollouts) in parallel and developing methods to effectively compare and select the most successful one. This paradigm shift recognizes that computer tasks often have multiple valid solution paths, and by exploring several approaches simultaneously, the chances of finding a successful solution increase dramatically.

At the heart of the bBoN framework lies the Behavior Narrative Generator, a novel component that transforms dense, information-heavy trajectories into compact, interpretable summaries. These “behavior narratives” capture the essence of what each agent did and how it affected the environment, filtering away irrelevant details while preserving task-relevant action-effect relationships. The generated narratives serve as an efficient representation that enables the second key component—the Behavior Best-of-N Judge—to perform comparative evaluation across multiple candidate solutions. By focusing on meaningful differences in behavior rather than raw screenshot sequences, the judge can make more informed decisions about which solution trajectory best accomplishes the task.

The researchers built their framework upon Agent S2, an existing open-source framework for computer-use agents. Additionally, they introduced Agent S3, an improved baseline that eliminates hierarchical planning in favor of a simpler flat policy that can replan at any time, and incorporates a coding agent that can execute programmatic edits when needed. This improved baseline alone established a new state-of-the-art result on the OSWorld benchmark, achieving 62.6% success rate on 100-step tasks. When combined with the bBoN framework to select from multiple rollouts, performance jumped to 69.9%, significantly outperforming previous methods and approaching the human benchmark of 72%.

Furthermore, the researchers demonstrated the framework’s versatility through comprehensive ablations and zero-shot generalization tests. The bBoN approach shows consistent improvements as the number of rollouts increases, validating the effectiveness of wide scaling. The researchers also investigated the impact of model diversity, finding that combining different capable models (like GPT-5 and Gemini 2.5 Pro) achieved the highest success rates, highlighting the value of diversity in solution approaches. When tested on different operating systems through the WindowsAgentArena and AndroidWorld benchmarks, bBoN demonstrated strong generalization, improving performance by 6.4% and 3.5% respectively, even with just three rollouts.

The implications of this research extend beyond benchmark improvements. Traditional approaches to enhancing AI systems have focused on making individual models more capable, but bBoN demonstrates that effectively combining multiple solution attempts can yield “unreasonable effectiveness” without requiring more sophisticated individual agents. The research suggests a promising direction for practical deployment of computer-use agents in real-world applications, where reliability and success rates are critical concerns.

The bBoN framework does have limitations, particularly in its assumption that agents can produce multiple independent rollouts from the same initial state. While this works well in research environments and controlled settings with virtual machines, applying the approach to a user’s actual desktop presents challenges due to potential interference between concurrent rollouts. Tasks that interact with shared online resources (like email or cloud storage) would need special handling to manage cross-run effects. The researchers acknowledge these constraints and suggest that future work could extend parallel rollouts to real desktops and better manage shared online resources.

In conclusion, “The Unreasonable Effectiveness of Scaling Agents for Computer Use” introduces a paradigm shift in how we think about improving computer-use agents. Rather than focusing exclusively on making individual agents smarter, the researchers demonstrate that intelligent scaling across multiple solution attempts can achieve remarkable results. By converting dense trajectories into compact behavior narratives and developing principled methods for trajectory selection, bBoN establishes a practical framework for improving the reliability and success rates of computer-use agents, bringing us significantly closer to the goal of robust, human-level automation of everyday digital tasks.