Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

Researchers introduce Emergence WebVoyager, an improved benchmark for evaluating web agents with standardized methodology. The study addresses critical gaps in existing AI agent evaluation practices, particularly for web-based tasks, by establishing clear guidelines for task definition, error handling, and reporting. Achieving 95.9% inter-annotator agreement, this framework enables more reliable performance comparisons. Testing on OpenAI Operator reveals 68.