How AI Impacts Skill Formation
Citation. Shen, J. H., & Tamkin, A. (2026). How AI Impacts Skill Formation. arXiv. https://arxiv.org/abs/2601.20245
Summary. Shen and Tamkin report randomized experiments on how AI assistance affects developers learning a new asynchronous Python library. In the main study, AI access did not significantly reduce completion time, but it reduced subsequent skill evaluation scores by 17% or about two grade points. The paper is useful for the wiki because it separates immediate task completion from the development of the conceptual understanding, code-reading, and debugging skills needed to supervise AI-produced work.
Key claims
- In the main randomized study with 52 participants, AI assistance reduced quiz performance by 4.15 points on a 27-point evaluation, a 17% difference or about two grade points (Cohen’s d = 0.738, p = 0.010), while task completion time did not improve significantly (p = 0.391).
- The evaluation covered conceptual understanding, code reading, and debugging; debugging questions showed the largest average score differences between AI and no-AI conditions.
- AI use was heterogeneous: low-scoring patterns involved full delegation, progressive reliance, or iterative AI debugging, while higher-scoring patterns involved conceptual inquiry, explanation-seeking, or generation followed by comprehension checks.
- Full AI delegation produced the fastest average completion time among the AI patterns, but with poor learning outcomes, while conceptual inquiry and hybrid explanation patterns preserved more skill formation by keeping the participant cognitively engaged.
- The authors argue that AI-supported productivity can mask reduced competence formation, especially where junior workers must later validate, debug, or supervise AI-generated work.
Evidence & limitations
- Evidence comes from pilot studies and a preregistered main randomized experiment using experienced Python programmers recruited through a crowd-work platform; the main study retained 52 participants balanced on coding experience, Python experience, prior asynchronous-programming familiarity, and AI coding-tool use.
- The task was intentionally bounded: participants learned and used the Trio Python library over a short session rather than developing workplace expertise over months or years.
- The AI condition used a chat-based assistant rather than a fully agentic coding tool; the authors treat this as a lower bound for cognitive offloading because more autonomous tools may require less human participation.
- The study measured skill formation through a post-task quiz rather than later job performance, transfer to new tasks, or longitudinal professional development.
- The paper is an arXiv preprint; the Drive PDF matched arXiv version 2 metadata and no publisher DOI was found in the arXiv record.