Search

The standard practice in machine learning for small-crystal property prediction relies on random train-test splitting of datasets such as the Materials Project. This approach creates an illusion of generalization: models routinely report near-zero mean absolute errors on formation energies, band gaps, or elastic constants, yet these impressive figures reflect leakage of structural, compositional, and energetic information rather than genuine out-of-distribution capability. Random splitting fails because crystals are not independent and identically distributed; repeated prototypes, shared elemental combinations, local coordination environments, and clustered formation energies ensure that train and test sets remain statistically entangled even after random partitioning. We identify a typology of four generalization illusions—prototype, compositional, energy-range, and structural—that systematically mislead the field and explain why published “state-of-the-art” accuracies collapse under more rigorous evaluation regimes. The consequences are severe: wasted experimental validation efforts, inflated claims of progress, overinvestment in architectures that cannot extrapolate, and a delayed recognition of fundamental limitations in current graph-network approaches. We propose six alternative evaluation strategies—composition splits, prototype splits, time splits, structural dissimilarity splits, energy-extrapolation splits, and cross-database splits—that replace random partitioning with deliberate distribution shifts, thereby restoring scientific integrity to benchmark design in computational materials science.

Benchmarking has become central to the development of machine learning interatomic potentials (MLIPs), yet the epistemic reliability of reported comparisons remains insufficiently scrutinized. This review synthesizes prevailing practices and demonstrates that current evaluation protocols systematically misrepresent model progress. A coherent taxonomy of methodological failure emerges, spanning opaque data handling, structurally flawed train–test partitioning, restricted metric selection, weak baseline construction, limited reproducibility, and the near absence of extrapolation analysis. Under these conditions, widely cited performance indicators—such as sub-10 meV/atom energy MAE or sub-0.1 eV/Å force RMSE—primarily capture interpolation within constrained training distributions, offering limited insight into generalization, dynamical stability, or deployment viability. A related deficiency lies in the systematic exclusion of physically and computationally salient regimes, including long-range interactions, finite-temperature behavior, low-symmetry and disordered structures, calibrated uncertainty, defect-rich configurations, and explicit cost–accuracy trade-offs. Existing benchmark suites, including Materials Project–derived datasets, QM9 adaptations, COMP6, and bespoke collections, inherit these constraints, reinforcing an evaluative paradigm that privileges narrow optimization over robust, application-relevant performance. Recasting benchmark outcomes as contingent on methodological design rather than intrinsic model capability reveals how evaluation choices implicitly structure model rankings. In response, this work advances a set of directly implementable standards: diversified splitting strategies, distribution-aware multi-metric reporting, transparent baseline inclusion, controlled extrapolation regimes, complete reproducibility artifacts, and normalized cost accounting. Aligning benchmarking practice with these principles is necessary to transition from incremental leaderboard gains toward reliable and transferable interatomic potentials for materials discovery.