The integration of artificial intelligence (AI) into materials science, often referred to as Materials AI, has revolutionized the field by enabling accelerated discovery, design, and optimization of new materials. This narrative review explores the governance and responsible use of Materials AI, focusing on standards, transparency, and risk management. Drawing on peer-reviewed literature, we examine the evolution of AI applications in materials science, ethical considerations, and the need for robust frameworks to ensure accountable deployment. Key themes include the ethical implications of data bias and intellectual property in AI-driven materials discovery; the development of standards for model validation and interoperability; mechanisms to enhance transparency in black-box AI models; and strategies to identify and mitigate risks, such as model unreliability and societal impacts. The review highlights how autonomous experimentation systems and machine learning techniques have transformed materials research, while underscoring the importance of reflexive governance to address potential harms. Objectives include synthesizing current practices, identifying gaps in responsible AI adoption, and proposing pathways for sustainable integration. By fostering transparency and risk-aware approaches, Materials AI can contribute to societal benefits, such as advancing energy materials and sustainable manufacturing, while minimizing ethical pitfalls. This work emphasizes the interdisciplinary nature of responsible Materials AI and calls for collaboration among scientists, policymakers, and ethicists to establish trustworthy systems.