Search

Materials informatics, the application of data science and machine learning to materials research, has revolutionized the discovery and design of new materials. However, the field faces significant challenges in reporting failure cases, negative results, and biases, which are often silenced in the literature. This review examines documented failures in materials informatics, such as data bias, model overoptimism, and reproducibility issues, and highlights the systemic factors that lead to their underreporting. Drawing on 30 recent peer-reviewed articles, we explore themes including data quality, algorithmic limitations, and publication bias. The objectives are to assess what is typically documented, identify silenced aspects, such as unsuccessful experiments, and propose strategies for more transparent reporting. By addressing these gaps, the review aims to foster a more robust and trustworthy materials informatics ecosystem, ultimately accelerating sustainable innovation in materials science.

This review systematically examines the treatment of absence and null results in the materials machine learning literature spanning 2017–2022, drawing exclusively on a curated set of 30 peer-reviewed publications and foundational works that address publication bias, negative findings, and reproducibility challenges in data-driven materials discovery. Through a targeted search strategy across databases such as Web of Science, Scopus, and arXiv using terms including “null result,” “negative result,” “publication bias,” “file drawer,” “failed synthesis,” and “reproducibility” combined with materials informatics keywords, the analysis reveals a persistent imbalance: while successful predictions and syntheses dominate published outputs, systematic documentation of failed predictions, unsuccessful syntheses, null correlations, and abandoned model architectures remains exceedingly rare. What is currently reported tends to be limited to negative outcomes that coincidentally reveal mechanistic insights or contradict high-profile hypotheses, whereas what is systematically unreported encompasses the vast majority of unsuccessful hyperparameter searches, negative active learning campaigns, and non-discoveries that yield no novel materials meeting target criteria. The typology of absence and null results developed here identifies six distinct categories—negative predictive outcomes, null hypothesis non-rejection, failed synthesis, non-discovery, failed replication, and abandoned architecture—each carrying unique implications for scientific progress. The consequences of this non-reporting include severe overestimation of model performance, widespread redundant experimental effort, a false sense of methodological consensus across the field, and slowed overall discovery rates as potentially informative negative signals remain invisible. Ultimately, this review offers concrete recommendations for authors, journals, and the broader community to shift incentives toward transparent reporting of absence, thereby restoring balance to the materials AI literature and accelerating reliable data-driven discovery.

Benchmarking has become central to the development of machine learning interatomic potentials (MLIPs), yet the epistemic reliability of reported comparisons remains insufficiently scrutinized. This review synthesizes prevailing practices and demonstrates that current evaluation protocols systematically misrepresent model progress. A coherent taxonomy of methodological failure emerges, spanning opaque data handling, structurally flawed train–test partitioning, restricted metric selection, weak baseline construction, limited reproducibility, and the near absence of extrapolation analysis. Under these conditions, widely cited performance indicators—such as sub-10 meV/atom energy MAE or sub-0.1 eV/Å force RMSE—primarily capture interpolation within constrained training distributions, offering limited insight into generalization, dynamical stability, or deployment viability. A related deficiency lies in the systematic exclusion of physically and computationally salient regimes, including long-range interactions, finite-temperature behavior, low-symmetry and disordered structures, calibrated uncertainty, defect-rich configurations, and explicit cost–accuracy trade-offs. Existing benchmark suites, including Materials Project–derived datasets, QM9 adaptations, COMP6, and bespoke collections, inherit these constraints, reinforcing an evaluative paradigm that privileges narrow optimization over robust, application-relevant performance. Recasting benchmark outcomes as contingent on methodological design rather than intrinsic model capability reveals how evaluation choices implicitly structure model rankings. In response, this work advances a set of directly implementable standards: diversified splitting strategies, distribution-aware multi-metric reporting, transparent baseline inclusion, controlled extrapolation regimes, complete reproducibility artifacts, and normalized cost accounting. Aligning benchmarking practice with these principles is necessary to transition from incremental leaderboard gains toward reliable and transferable interatomic potentials for materials discovery.

Materials AI stands at a critical juncture where rapid advances in machine learning and data-driven discovery are reshaping materials engineering. Yet the field is severely constrained by a deep-rooted publication bias that systematically favors positive results while suppressing negative outcomes. This position paper asserts that Materials AI urgently requires widespread failure reporting and that negative results must be recognized and maintained as essential community infrastructure. In the prevailing culture, only successful predictions, high-accuracy models, and newly discovered materials reach publication, while failed models, insufficient datasets, misleading benchmarks, and unsuccessful synthesis attempts remain hidden. The consequences are profound: researchers repeatedly traverse the same unproductive paths, the scientific literature presents a distorted view of reality, meta-analysis becomes impossible, and overall progress slows dramatically. Failure reporting functions as shared infrastructure—much like open databases or standardized protocols—by mapping the boundaries of what does not work and thereby enabling more efficient exploration by the entire community. Four core types of failures demand routine documentation: model failures, data failures, benchmark failures, and synthesis failures, with additional emphasis on transfer and extrapolation failures. By treating negative results as valuable contributions rather than career liabilities, the community can accelerate discovery, enhance reproducibility, and build a more trustworthy knowledge base. This paper articulates the position, analyzes the publication bias problem, defines failure reporting as infrastructure, classifies failure types, and identifies barriers before offering concrete recommendations. Embracing negative results will transform Materials AI from a literature of selective successes into a robust, transparent, and collectively intelligent discipline capable of delivering sustainable materials innovation at scale.