Search

This review systematically surveys conceptual approaches to scientific validation in artificial intelligence applications for materials science, drawing exclusively on 50 peer-reviewed publications from 2017 to 2022 to examine how validation is defined, operationalized, critiqued, and innovated upon within the domain. The methodology followed a targeted literature search protocol across Web of Science, Scopus, and arXiv using eight predefined search strings focused on validation, cross-validation, out-of-distribution testing, generalization, and related terms in materials AI, with strict inclusion criteria requiring explicit discussion of conceptual or epistemological aspects of validation and exclusion of purely empirical performance reports, ultimately yielding the 50 selected references after PRISMA-style screening of approximately 250 unique records. Current validation practices in materials AI literature remain anchored in conventional statistical techniques such as random train-test splits, k-fold cross-validation, leave-one-out cross-validation, and hold-out test sets, which the surveyed papers predominantly employ to quantify predictive accuracy on materials property prediction, discovery, and design tasks. Critical findings demonstrate that these practices frequently claim to establish reliable generalization while actually capturing only in-sample performance, systematically overlooking hidden data structures, distribution shifts, feature selection leakage, and the small-data regimes intrinsic to materials science, thereby producing inflated estimates of model utility that do not translate to real-world deployment. To structure the field’s understanding, the review advances a taxonomy of validation approaches organized hierarchically by what they seek to validate—predictive accuracy, robustness, generalizability, and causal structure—providing a conceptual scaffold for aligning methods with task-specific requirements. Recommendations emphasize explicit reporting, justification of method choice, and community-wide benchmarks. At the same time, open challenges persist in areas such as validating generative models for novelty and enabling trustworthy extrapolation beyond training distributions, underscoring an urgent need for epistemologically grounded practices that match the high-stakes demands of materials discovery.