Super Mario Meets AI: Experimental Effects of Automation and Skills on Team Performance and Coordination
Citation. Dell’Acqua, F., Kogut, B., & Perkowski, P. (2025). Super Mario Meets AI: Experimental Effects of Automation and Skills on Team Performance and Coordination. The Review of Economics and Statistics, 107(4), 951–966. https://doi.org/10.1162/rest_a_01328
Summary. Dell’Acqua, Kogut, and Perkowski report a laboratory experiment in which four-person “firms” played a team-based Nintendo Switch coordination game and some teams had a human player replaced by a high-performing automated agent. Even though the AI agent outperformed humans individually, automation reduced overall team performance, increased coordination failures, and lowered trust and effort among directly affected human teammates. The paper is useful for the wiki because it shows that the unit of AI impact is not only task capability but the social and coordination system into which automation is introduced.
Key claims
- Introducing high-performing automated agents reduced firm performance: across Phase 3, AI firms returned 0.51 fewer ingredients per team per round than control firms, equivalent to 6.12 fewer ingredients across the two teams over six rounds and about a 13% lower performance bonus.
- The performance penalty was largest immediately after automation: in round 7, AI-assigned teams returned about one fewer ingredient than control teams, an almost 8% decrease relative to the control mean, before the gap narrowed and disappeared by the final round.
- Automation operated through channels different from ordinary human turnover: short-term performance fell in both the team receiving the AI and the spillover team, whereas new human hires mainly affected the directly changed team.
- Automation increased coordination failures even though the AI agent was highly capable: AI firms averaged 1.28 more coordination failures per round in Phase 3, and directly affected AI teams averaged 2.81 more failures per round.
- Directly affected players reported lower trust and effort after working with the automated partner, with trust 1.44 standard deviations lower and effort 0.81 standard deviations lower, while general attitudes toward AI did not significantly change.
- Automation effects were skill-biased at the team level: low- and medium-skilled remaining teams experienced performance declines, while high-skilled teams did not, suggesting that AI complementarity depended on the skills of the humans who had to integrate the automated coworker.
Evidence & limitations
- Evidence comes from a preregistered laboratory experiment using 220 participants grouped into 55 four-person firms, with random assignment to control, new-hire, or AI replacement conditions after teams had already developed coordination routines.
- The task was a stylized team game rather than workplace production, but it was designed to isolate coordination, routines, effort, trust, and skill complementarity under human-machine team change.
- The “AI” was an automated game agent in a bounded coordination task, not a generative AI system or autonomous workplace agent; the source is strongest as evidence about automation entering interdependent team routines rather than about language-model collaboration specifically.
- The study measured short-run adaptation over repeated rounds and post-task attitudes, not durable workplace adoption, long-term motivation, or organizational redesign.