Scaffolding Human–AI Collaboration: A Field Experiment on Behavioral Protocols and Cognitive Reframing
Citation. Farach, A., Cambon, A., Tankelevitch, L., Hsueh, C., & Janssen, R. (2026). Scaffolding Human–AI Collaboration: A Field Experiment on Behavioral Protocols and Cognitive Reframing. arXiv. https://arxiv.org/abs/2604.08678
Summary. In a single-day field experiment with 388 Gap Inc. employees, Farach et al. compared a required process for pairs to use Copilot together with unstructured use, and compared individual thought-partner training with standard feature training. The required pair process was associated with lower AI-graded document quality and far fewer completed documents. Thought-partner training did not improve average quality on the study’s main test, although a later, less conclusive analysis suggested that trainees may have been more likely to receive a perfect score. These findings are uncertain because dropout differed between groups, morning and afternoon sessions were not fully comparable, and an AI system graded the work.
Key claims
- In Task A, pairs assigned to the Create-Out-Loud process scored 4.96 points lower on the 22-point AI-graded document-quality scale than pairs using Copilot without that structure (10.68 vs. 15.63; 95% CI [-6.71, -3.20]; p < .001). The estimated harm remained even after a conservative adjustment for the groups’ different dropout rates.
- The behavioral-protocol condition had substantially lower odds of producing a Task A document (OR = 0.12, 95% CI [0.04, 0.35]; p < .001); 26.8% of treatment pairs versus 4.1% of control pairs failed to produce one.
- In Task B, thought-partner training did not produce a statistically reliable improvement in average quality on the study’s main test (18.1 vs. 17.3; p = .223). Because 68.1% of participants received the maximum score, the test had little room to detect improvement. A later analysis found that trained participants were more likely to receive a perfect score (OR = 2.07, 95% CI [1.12, 3.83]; p = .022), but this result was not part of the original analysis plan and did not remain reliable under a conservative adjustment for dropout.
- The training group reported larger gains in exploration-and-experimentation beliefs (BH-adjusted p = .013) and in the combined belief measure (BH-adjusted p = .047). However, a different statistical adjustment found no reliable difference, suggesting that the apparent gain may partly reflect recovery from the earlier task rather than the training itself.
Evidence & limitations
- Public arXiv v2 preprint reporting a randomized, two-arm field experiment at one Fortune 500 retailer; the public arXiv PDF exactly matches the ingested Drive binary (SHA-256: 6caa83c874b308ff3b7b8338470a75f3251ad28ebb9022329e96bf4814cf8359).
- The behavioral and cognitive interventions differ in unit of analysis, task structure, outcome saturation, attrition, and sequence, so the study does not cleanly establish a general behavioral-versus-cognitive scaffolding taxonomy.
- Treatment assignment was confounded with AM/PM session timing; treatment conditions also had higher attrition, and LLM-as-judge scores were strongly associated with word count for Task A.
- The extracted PDF contained one ordinary scholarly reference to large language models and no model-directed or ingest-manipulation instructions.