Are We Overfitting to the Materials Project? A Critique of Dataset Bias in Property Prediction Benchmarks

iamrp.net-z8UX-1791491021-v815034582 1791491021 Institute for Advanced Materials Research Press info@iamrp.net Institute for Advanced Materials Research Press Journal of Computational and Data-Driven Materials Engineering J. Comput. Data-Driven Mater. Eng. 3149-9368 2025 4 1 Are We Overfitting to the Materials Project? A Critique of Dataset Bias in Property Prediction Benchmarks Ricardo Alves Bruno Costa Helena Martins Nuno Faria 2025 10.68159/v815034582 https://iamrp.net/pub/journal/2/article/v815034582 Zivic F, Malisic AK, Grujovic N, Stojanovic B, Ivanovic M. Materials informatics: A review of AI and machine learning tools, platforms, data repositories, and applications to architectured porous materials. Mater Today Commun. 2025;48:113525. 10.1016/j.mtcomm.2025.113525 Ward L, Dunn A, Faghaninia A, Zimmermann NE, Bajaj S, Wang Q, et al. Matminer: An open source toolkit for materials data mining. Comput Mater Sci. 2018;152:60-9. 10.1016/j.commatsci.2018.05.018 Davariashtiyani A, Wang B, Hajinazar S, Zurek E, Kadkhodaei S. Impact of data bias on machine learning for crystal compound synthesizability predictions. Mach Learn Sci Technol. 2024;5(4):040501. 10.1088/2632-2153/ad9378 Hu J, Liu D, Fu N, Dong R. Realistic material property prediction using domain adaptation based machine learning. Digit Discov. 2024;3(2):300-12. 10.1039/D3DD00162H Lee EH, Jiang W, Alsalman H, Low T, Cherkassky V. Methodological framework for materials discovery using machine learning. Phys Rev Mater. 2022;6(4):043802. 10.1103/PhysRevMaterials.6.043802 Chan CH, Sun M, Huang B. Application of machine learning for advanced material prediction and design. EcoMat. 2022;4(4):e12194. 10.1002/eom2.12194 Xie T, Grossman JC. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys Rev Lett. 2018;120(14):145301. 10.1103/PhysRevLett.120.145301 Chen C, Ye W, Zuo Y, Zheng C, Ong SP. Graph networks as a universal machine learning framework for molecules and crystals. Chem Mater. 2019;31(9):3564-72. 10.1021/acs.chemmater.9b01294 Kumagai M, Ando Y, Tanaka A, Tsuda K, Katsura Y, Kurosaki K. Effects of data bias on machine-learning-based material discovery using experimental property data. Sci Technol Adv Mater Methods. 2022;2(1):302-9. 10.1080/27660400.2022.2109447 Li K, DeCost B, Choudhary K, Greenwood M, Hattrick-Simpers J. A critical examination of robustness and generalizability of machine learning prediction of materials properties. npj Comput Mater. 2023;9(1):55. 10.1038/s41524-023-01012-9 Li K, Persaud D, Choudhary K, DeCost B, Greenwood M, Hattrick-Simpers J. Exploiting redundancy in large materials datasets for efficient machine learning with less data. Nat Commun. 2023;14(1):7283. 10.1038/s41467-023-42992-y Tilley RJD. Crystals and crystal structures. 2nd ed. Hoboken (NJ): John Wiley & Sons; 2020. Xu P, Ji X, Li M, Lu W. Small data machine learning in materials science. npj Comput Mater. 2023;9(1):42. 10.1038/s41524-023-01000-z Goodall REA, Lee AA. Predicting materials properties without crystal structure: Deep representation learning from stoichiometry. Nat Commun. 2020;11(1):6280. 10.1038/s41467-020-19964-7 Wang T, Zhang C, Snoussi H, Zhang G. Machine learning approaches for thermoelectric materials research. Adv Funct Mater. 2020;30(5):1906041. 10.1002/adfm.201906041 Jha D, Choudhary K, Tavazza F, Liao WK, Choudhary A, Campbell C, et al. Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning. Nat Commun. 2019;10(1):5316. 10.1038/s41467-019-13297-w Fung V, Zhang J, Juarez E, Sumpter BG. Benchmarking graph neural networks for materials chemistry. npj Comput Mater. 2021;7(1):84. 10.1038/s41524-021-00554-0 Li Q, Miklaucic N, Hu J. Out-of-distribution material property prediction using adversarial learning. J Phys Chem C. 2025;129(13):6372-85. 10.1021/acs.jpcc.4c07481 Singhal P, Walambe R, Ramanna S, Kotecha K. Domain adaptation: Challenges, methods, datasets, and applications. IEEE Access. 2023;11:6973-7020. 10.1109/ACCESS.2023.3237025 Cano AV, Payne JL. Mutation bias interacts with composition bias to influence adaptive evolution. PLoS Comput Biol. 2020;16(9):e1008296. 10.1371/journal.pcbi.1008296 Li Q, Fu N, Omee SS, Hu J. MD-HIT: Machine learning for material property prediction with dataset redundancy control. npj Comput Mater. 2024;10(1):245. 10.1038/s41524-024-01426-z Karton A, De Oliveira MT. Good practices in database generation for benchmarking density functional theory. Wiley Interdiscip Rev Comput Mol Sci. 2025;15(1):e1737. 10.1002/wcms.1737 Momennejad I, Sinclair S, Cikara M. Computational justice: Simulating structural bias and interventions. bioRxiv [Preprint]. 2019:776211. 10.1101/776211 Krishnan R, Sinha A, Ahuja N, Subedar M, Tickoo O, Iyer R. Mitigating sampling bias and improving robustness in active learning. arXiv [Preprint]. 2021. 10.48550/arXiv.2109.06321 Arjovsky M. Out of distribution generalization in machine learning [dissertation]. New York: New York University; 2020. Miceli M, Posada J, Yang T. Studying up machine learning data: Why talk about bias when we mean power? Proc ACM Hum-Comput Interact. 2022;6(GROUP):34. 10.1145/3492853 Schmidt J, Marques MRG, Botti S, Marques MAL. Recent advances and applications of machine learning in solid-state materials science. npj Comput Mater. 2019;5(1):83. 10.1038/s41524-019-0221-0 Wang AYT, Murdock RJ, Kauwe SK, Oliynyk AO, Gurlo A, Brgoch J, et al. Machine learning for materials scientists: An introductory guide toward best practices. Chem Mater. 2020;32(12):4954-65. 10.1021/acs.chemmater.0c01907 Rabanser S, Günnemann S, Lipton Z. Failing loudly: An empirical study of methods for detecting dataset shift. Adv Neural Inf Process Syst. 2019;32:1394-406.