Algorithmic assessment

Algorithmic assessment is not just selection with newer software. It is a measurement chain: a signal is collected, transformed by a model, interpreted as a construct, and used in an employment decision. The useful question is not whether “AI hiring tools” are good or bad in general. It is what construct is being inferred, from what signal, by what model, for what decision, under what evidence and governance standard.

2016-chamorro-premuzic-new-talent-signals gives the foundational I-O caution: new talent signals may be technologically novel, but they still need theory, construct clarity, validity evidence, and comparison against established methods. 2023-landers-machine-learning-psychometric-assessment sharpens that caution empirically. Machine learning does not automatically improve conventional psychometric selection batteries, and its clearest value may appear when assessment design changes, not when regression is simply swapped out for a more complex algorithm. 2023-nyc-automated-employment-decision-tools-faq adds the governance layer by defining covered AEDTs, bias audits, independent auditors, impact ratios, and notice obligations.

The topic sits at the intersection of construct-validity, evidence-based-management, and automation-and-substitution. Construct validity asks whether the tool is measuring a coherent job-relevant attribute rather than a convenient proxy. Evidence-based management asks whether the validity, utility, audit, or compliance claim is supported by evidence matched to the decision. Automation and substitution asks whether the tool supports human judgment, substantially assists it, or replaces discretionary decision-making in screening, ranking, or selection.

2022-strah-diversity-issues-job-analysis adds a deeper foundation problem for algorithmic assessment. Even before signals are modeled or audited, the job analysis that defines the criterion domain may miss underrepresented workers’ tasks, constraints, voice, or successful performance strategies. That matters for algorithmic tools because a model can be technically validated against a criterion that already encodes an incomplete or majority-normed account of the job.

2022-dellacqua-falling-asleep-at-wheel adds the human-oversight problem inside algorithmic hiring. In a field experiment with professional recruiters, algorithmic recommendations improved accuracy overall, but higher-stated AI quality made recruiters more likely to follow recommendations and lower-quality AI induced more effort and time spent reviewing resumes. For assessment, this means the validity of a human-AI selection system depends not only on model accuracy, but on how recommendations reshape human attention, discretion, and willingness to override.

2025-matic-balancing-accuracy-acceptance-algorithmic-hiring focuses on a related design question: how a hiring system combines a decision maker’s judgment with other applicant information. Across two experiments using predictions for real applicants, letting people make an intuitive overall judgment and then adding it to the model as a fixed, explicit input was more accurate than either intuitive judgment alone or freely editing the model’s final recommendation. The study’s practical implication is not that human input is automatically valuable. It is that a system can preserve a limited, visible human role without treating each exception as a reason to override the same evidence differently.

2025-landers-ethical-use-ai-iop adds the professional-ethics frame around AI selection. AI-scored or AI-delivered interviews are not only technical prediction problems: they require disclosure, evidence about how new data formats are collected and interpreted, attention to avatar or delivery-system effects, consistency across applicants, and scrutiny of foundational models and customization processes. This links algorithmic assessment to responsible-ai-deployment because selection AI can satisfy an efficiency demand while still failing disclosure, job-relatedness, fairness, accountability, applicant-experience, or domain-specific validation standards.

Two narrative recruitment-fairness articles converge that governance limits must extend beyond the final decision rule: historical data, labels, feature choices, model design, and evaluation can each preserve inequity, and the proposed safeguards include representative data, audits, explainability, regulatory attention, and meaningful human oversight 2025-agbasiere-algorithmic-fairness-recruitment 2025-ahuchogu-ai-bias-recruitment. Their convergence identifies a governance agenda, not validation evidence: neither article reports a controlled validation of a hiring model or a mitigation intervention 2025-agbasiere-algorithmic-fairness-recruitment 2025-ahuchogu-ai-bias-recruitment.

Connections

• Relates to construct-validity because algorithmic assessment only matters if the inferred construct is coherent, job-relevant, and defensible in a nomological network.
• Relates to evidence-based-management because prediction, validity, adverse impact, compliance, and utility are different evidence claims and should not be collapsed into “the tool works.”
• Relates to automation-and-substitution because AEDTs can support, substantially assist, or replace discretionary judgment, including at the screening stage before any final employment decision.
• Relates lightly to human-ai-collaboration when algorithmic outputs are used as decision support; the NYC AEDT language is a reminder that some “support” tools cross into substantial assistance or replacement.
• Relates to automation-complacency because assessment systems can preserve formal human discretion while still making workers inattentive followers of model advice.
• Relates to job-analysis because algorithmic assessment inherits its target definition from upstream work analysis; bad job analysis can make later model validation look cleaner than it is.
• Relates to responsible-ai-deployment because selection AI turns deployment ethics into concrete questions about disclosure, validation, fairness, applicant experience, appealability, and accountability; the two narrative recruitment-fairness articles additionally frame audits, explainability, and human oversight as governance safeguards rather than substitutes for validation 2025-agbasiere-algorithmic-fairness-recruitment 2025-ahuchogu-ai-bias-recruitment.
• Relates to inclusive-hr-systems because recruitment fairness must address potential exclusion in historical data, labels, features, model design, evaluation, and human use of the system rather than treating the model as the only source of risk 2025-agbasiere-algorithmic-fairness-recruitment 2025-ahuchogu-ai-bias-recruitment.

Contradictions & open questions

• The cluster separates prediction from explanation. 2016-chamorro-premuzic-new-talent-signals argues that understanding constructs matters, while some big-data assessment practices prioritize predictive relationships without enough theory.
• 2023-landers-machine-learning-psychometric-assessment suggests machine learning’s value may be limited in conventional scale-composite batteries but stronger in item-level or novel-data settings. That creates a design question: is the organization improving assessment, or merely adding model complexity?
• 2023-nyc-automated-employment-decision-tools-faq makes bias audits and notices central to compliance, but policy compliance does not by itself establish construct validity, job relatedness, or decision utility.
• “Decision support” is not a safe category by default. A simplified score, classification, or ranking can substantially assist or replace discretion even when a human remains nominally accountable for the final decision.
• 2022-dellacqua-falling-asleep-at-wheel complicates the idea that better hiring AI straightforwardly improves human decisions. Higher-quality recommendations can increase reliance and reduce effort, so the assessment unit is the coupled human-AI decision process, not the model alone.
• 2025-matic-balancing-accuracy-acceptance-algorithmic-hiring finds that a structured way of retaining human input can improve prediction and participant acceptance, but its simulated selection task does not show whether the same arrangement changes recruiter behavior, applicant experience, fairness, or legal defensibility in live hiring.
• Bias audits often focus on output disparities, but 2022-strah-diversity-issues-job-analysis suggests another layer: whether the criterion and job requirements being predicted already excluded some groups’ real work contributions.
• 2025-agbasiere-algorithmic-fairness-recruitment and 2025-ahuchogu-ai-bias-recruitment converge on audits, explainability, representative data, and human oversight, but neither tests whether those safeguards reduce disparate impact, establish job-relatedness, or improve selection utility in a live hiring system.