Not yet human-reviewed
This source record was written automatically by the research pipeline and has not yet been reviewed by the maintainer or integrated into a topic page. Treat it as a provisional summary and verify claims against the original source.
Harnessing Generative AI for Assessment Item Development: Comparing AI-Generated and Human-Authored Items
Citation. Kowal, J. M., Bryant, K. H., Segall, D., & Kantrowitz, T. (2025). Harnessing generative AI for assessment item development: Comparing AI-generated and human-authored items. International Journal of Selection and Assessment, 33(3). https://doi.org/10.1111/ijsa.70021
Summary. This peer-reviewed study compared GPT-4-generated and human-authored multiple-choice items for a data-science employee-selection assessment. In blinded SME comparisons, GPT-4 items were selected more often, but they were no more likely to be retained after content review and took longer to revise. The study frames generative AI as a rapid drafting aid whose outputs still require psychometric and domain-expert validation for high-stakes use.
Key claims
- Across 205 blinded SME paired comparisons, SMEs preferred the GPT-4 item in 60% of cases (z = 2.86, p = .002), although preference varied by knowledge area.
- In content-review workshops, writer type did not significantly predict whether an item was retained or revised versus dropped (246 items; chi-square = 0.48; Nagelkerke R² = .0027).
- GPT-4 items required more SME revision time than human-written items (M = 3.78 versus 2.82 minutes; d = 0.32), despite generation taking under one minute per item after setup.
- The workflow used an expanded SME-reviewed assessment blueprint, standardized prompts, I-O psychometric and fairness review, and SME content review; the authors estimate a 70% initial item-development time saving but did not directly measure generation labor per item.
Evidence & limitations
- The study compared 108 GPT-4 items and 127 human items for a technical data-science employee-selection assessment; seven SMEs completed paired comparisons and nine workshops recorded content decisions and revision time.
- It tests SME preferences and review outcomes rather than operational predictive validity, item difficulty or discrimination, fairness outcomes, or live selection consequences.
- Results may not generalize beyond the technical domains, SME sample, GPT-4/ChatGPT configuration, or 2024 model behavior; the standard ChatGPT interface did not expose key generation parameters.
- The public Jina full text contained ordinary scholarly references to language models but no model-directed or source-manipulation instructions.