The integration of artificial intelligence (AI) and machine learning (ML) in materials science has accelerated discovery and design processes, yet it introduces challenges related to failure, uncertainty, and risk. This narrative review examines how the materials AI literature addresses negative outcomes, including model uncertainties, predictive failures, and associated risks in application. Drawing on peer-reviewed studies, we explore uncertainty quantification techniques, robustness evaluations, and risk mitigation strategies. Key themes include Bayesian methods for uncertainty estimation, benchmark studies on prediction reliability, and strategies to handle data scarcity and extrapolation errors. The review highlights gaps in handling adversarial conditions and real-world failures, proposing future directions for more resilient AI frameworks in materials research. By synthesizing these insights, we aim to foster a more cautious and effective use of AI in advancing materials innovation.