Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters
Citation. Dell’Acqua, F. (2022). Falling asleep at the wheel: Human/AI collaboration in a field experiment on HR recruiters. Working paper.
Summary. This paper combines a formal model with a preregistered field experiment on AI-supported resume screening. Dell’Acqua argues that as AI advice becomes more accurate, human workers may reduce effort and attention, allowing the AI to substitute for rather than augment their judgment. In the experiment, professional recruiters reviewed resumes with no AI, low-quality AI, high-quality AI, or a perfectly predictive benchmark. The central design implication is that the best standalone algorithm is not always the best human-AI system if it changes human effort, learning, and oversight.
Key claims
- In a field experiment, 181 professional recruiters collectively reviewed nearly 8,000 resumes for a software-engineering position and made callback decisions based on math-relevant applicant information.
- Recruiters assigned to AI advice followed recommendations in proportion to stated AI quality: about 84.9% for the perfectly predictive benchmark, 73.9% for the high-performing AI, and 53.1% for the lower-performing AI.
- Pooling AI conditions, AI assistance improved recruiter accuracy relative to no AI; however, among the imperfect AI conditions, higher-quality AI induced more reliance while lower-quality AI induced more effort and more time spent evaluating resumes.
- Recruiters receiving lower-quality AI spent about 8.8–10.0 seconds more per resume than controls in the reported time-spent models, while high-quality AI did not increase time spent.
- The evidence supports an automation-complacency mechanism: improving AI advice can reduce human effort and learning, so combined human-AI performance depends on behavioral response, not only model accuracy.
- The strongest learning and effort effects were driven by more experienced recruiters, suggesting that worker skill can interact with AI quality in non-linear ways.
Evidence & limitations
- Design: preregistered field experiment with professional recruiters hired through an online freelancer platform. Recruiters were randomly assigned to no AI, a perfectly predictive benchmark, high-performing AI, or lower-performing AI, and evaluated applicant profiles constructed from OECD PIAAC data with objective math-performance measures.
- Evidence strength: behavioral field-experimental evidence on decision accuracy, AI-following, time spent, clicks/effort, and learning over repeated evaluations in an HR selection-like task.
- Limitations: working-paper version with public provenance less stable than a journal DOI; the OSF preregistration is described as embargoed in the PDF; the task uses constructed applicant profiles and a single math-ability criterion, so it does not represent the full construct validity or adverse-impact problem in real hiring; the “perfect prediction” condition is an unrealistic benchmark rather than a deployable system.