How Well Does Agent Development Reflect Real-World Work?
Citation. Wang, Z. Z., Vijayvargiya, S., Chen, A., Zhang, H., Arangarajan, V. A., Chen, J., Chen, V., Yang, D., Fried, D., & Neubig, G. (2026). How Well Does Agent Development Reflect Real-World Work? arXiv. https://arxiv.org/abs/2603.01203
Summary. Wang and colleagues map AI-agent benchmark tasks into O*NET-derived work-domain and work-activity taxonomies to test whether agent development reflects the structure of human work. Across 43 benchmarks, 72,342 tasks, and 1,016 U.S. occupations, they find that benchmark effort is concentrated in programming-heavy computer and mathematical work while many highly digitized and economically important domains remain underrepresented. The paper also proposes task-complexity and autonomy measures to turn benchmark performance into a more practical account of which work agents can complete end-to-end at different complexity levels.
Key claims
- Existing agent benchmarks collectively cover 56.5% of the domain taxonomy and 85.4% of the skill taxonomy, suggesting broader coverage of general work activities than occupation-specific contexts.
- Agent benchmarking effort overrepresents computer and mathematical work, a domain the authors report as 7.6% of U.S. employment, while management, legal, and architecture/engineering have high digital-work ratios but low representation among benchmark examples.
- At the skill level, current agent development disproportionately targets a small set of fine-grained skills accounting for less than 5% of U.S. employment, while interpersonal interaction and other widely distributed work activities are comparatively absent.
- The authors define agent autonomy as a performance frontier over increasing task complexity, arguing that automation and augmentation should be treated as scope-dependent positions on a spectrum rather than as mutually exclusive categories.
- The paper recommends three benchmark-design principles for work-relevant agents: improve domain and skill coverage, ensure realism and complexity in task generation, and evaluate performance granularly across intermediate workflow steps rather than only final task success.
Evidence & limitations
- The analysis uses 43 agent benchmarks, a sampled/mapped set of 72,342 task instances, O*NET work taxonomies, BLS employment and wage data, and LLM-assisted task annotation with manual verification.
- The authors report that manually verified LM mappings aligned with human judgment 92% of the time for domain paths and 93% for skill paths, but remaining disagreement is expected when benchmark task descriptions are underspecified.
- The source is an arXiv preprint; the Drive PDF is version 1 from 2026-03-01, while the arXiv API currently reports version 2 updated 2026-03-06, so stable provenance should use the unversioned arXiv record while preserving the downloaded file hash.
- The study evaluates benchmark representativeness and capability measurement, not realized organizational adoption, job redesign, employment change, or worker experience after agent deployment.