Not yet human-reviewed
This source record was written automatically by the research pipeline and has not yet been reviewed by the maintainer or integrated into a topic page. Treat it as a provisional summary and verify claims against the original source.
Vibe Coding: An Experiment with Test-Driven Development
Citation. Mock, M. & Russo, B. (2026). Vibe Coding: An Experiment with Test-Driven Development. arXiv:2607.22406.
Summary. This exploratory preprint compares solo, collaborative human–CLLM, fully automated, and agentic test-driven-development workflows on software-artifact quality and process efficiency. In a controlled pre-experimental study of 16 TDD-experienced professionals plus repeated automated executions, agentic workflows achieved the highest average functional correctness but produced more complex, less completely tested code, while collaborative workflows produced test suites closer to the baseline organization.
Key claims
- The agentic workflow averaged 94.58% test-passing rate and differed significantly from the other workflows on functional correctness after Holm–Bonferroni correction, although human-involving workflows showed greater variability.
- Agentic outputs had more complex control flow and baseline-test branch coverage of at most 75%, whereas some solo and collaborative participants reached 100% branch coverage.
- Collaborative workflow participants produced test-suite organization closer to the baseline suite than fully automated or agentic workflows, but iterative CLLM interaction imposed coordination overhead.
Evidence & limitations
- The study combines a controlled pre-experimental comparison of solo and collaborative workflows with repeated executions for fully automated and agentic workflows, using the same tasks and quality metrics but different evaluation settings.
- It uses two simple Python tasks, a limited professional sample, one CLLM for the human-centred comparison, and implementation-specific configurations; the authors describe the findings as exploratory.
- Public provenance was verified against the arXiv landing page and PDF; the public PDF SHA-256 exactly matches the ingested Drive artifact.