Search

The standard practice in machine learning for small-crystal property prediction relies on random train-test splitting of datasets such as the Materials Project. This approach creates an illusion of generalization: models routinely report near-zero mean absolute errors on formation energies, band gaps, or elastic constants, yet these impressive figures reflect leakage of structural, compositional, and energetic information rather than genuine out-of-distribution capability. Random splitting fails because crystals are not independent and identically distributed; repeated prototypes, shared elemental combinations, local coordination environments, and clustered formation energies ensure that train and test sets remain statistically entangled even after random partitioning. We identify a typology of four generalization illusions—prototype, compositional, energy-range, and structural—that systematically mislead the field and explain why published “state-of-the-art” accuracies collapse under more rigorous evaluation regimes. The consequences are severe: wasted experimental validation efforts, inflated claims of progress, overinvestment in architectures that cannot extrapolate, and a delayed recognition of fundamental limitations in current graph-network approaches. We propose six alternative evaluation strategies—composition splits, prototype splits, time splits, structural dissimilarity splits, energy-extrapolation splits, and cross-database splits—that replace random partitioning with deliberate distribution shifts, thereby restoring scientific integrity to benchmark design in computational materials science.