Search

Machine learning interatomic potentials have become central tools in computational materials engineering, promising accurate and scalable predictions of alloy properties. Benchmarks in the field routinely report impressive accuracies, such as energy mean absolute errors below 20 meV/atom and low force root-mean-square errors, leading many papers to conclude that their models generalize well to alloy systems. Yet when these same potentials are deployed on real-world disordered alloys—such as binary solid solutions or multi-principal-element high-entropy alloys—the accuracy collapses, often by an order of magnitude. The root cause lies in the benchmarking pipeline itself: training and testing occur almost exclusively on ordered crystal structures drawn from databases of intermetallic compounds, while disordered configurations, which dominate practical applications, are never evaluated. This critical critique identifies four primary sources of systematic overestimation. First, the near-universal reliance on ordered training and test sets creates an illusion of generalization that fails the moment configurational disorder is introduced. Second, random or composition-based splits on ordered data allow models to memorize recurring structural motifs rather than learn transferable physics. Third, even when special quasirandom structures are employed as proxies for disorder, their periodic nature and limited sampling of local environments produce a misleading “SQS mirage.” Fourth, energy-only metrics mask severe force-prediction failures that render molecular-dynamics simulations unstable in disordered systems. These benchmarking flaws have concrete consequences: overconfidence among practitioners, misallocation of experimental resources, and slowed progress in high-entropy alloy design. By drawing on the literature, this article demonstrates that the ordered-to-disordered gap is not an edge case but a fundamental and under-reported limitation. It proposes concrete reforms—mandatory disordered test sets, configurational sampling protocols, and separate reporting of ordered versus disordered performance—to restore credibility to ML potential benchmarks. Until these changes are adopted, claims of “alloy-ready” potentials should be treated with skepticism. The field must move beyond ordered-crystal comfort zones if machine-learned potentials are to deliver on their promise for the compositionally complex materials that define modern materials science.

High-entropy alloys (HEAs) and multi-principal element alloys (MPEAs) occupy an enormous compositional space that conventional computational approaches cannot fully explore. A five-element system with 10 % concentration steps already contains millions of distinct nominal compositions, each further multiplied by exponentially many atomic configurations arising from configurational disorder. Machine learning (ML) interatomic potentials have been proposed as a scalable solution to accelerate property prediction and materials design in this vast space. Yet the compositional generalizability of these potentials—their capacity to make reliable predictions for compositions and local environments lying outside the training distribution—remains largely unexamined in a systematic way. Most existing benchmarks focus on interpolation within narrow ranges of ordered or equiatomic compounds, leaving critical extrapolation scenarios untested. This conceptual framework article identifies four distinct but interrelated dimensions of compositional generalizability in HEA ML potentials: (i) element extrapolation, (ii) concentration interpolation, (iii) multi-element recombination, and (iv) local environment diversity. It proposes a four-component assessment framework—training-set characterization, test-set design, dimension-specific generalization metrics, and a structured validation protocol—that enables researchers to quantify generalization gaps without relying on performance numbers or simulation results. For each dimension, explicit validation strategies and success criteria are defined, grounded in the literature on special quasirandom structures, graph-network representations, and equivariant architectures. The framework is deliberately conceptual, emphasizing definitions, relationships among components, assessment criteria, and diagnostic reasoning rather than empirical data. By adopting this framework, the community can move beyond ad-hoc testing and develop ML potentials that truly generalize across the high-dimensional alloy landscape. The implications extend to more reliable high-throughput screening, accelerated discovery of novel HEAs, and clearer guidance for training-set design and model architecture choices. Ultimately, systematic assessment of compositional generalizability will help ensure that ML potentials fulfill their promise for complex concentrated alloys.