Diffusion-based generative models have become a leading approach in artificial intelligence for sampling complex data distributions and have been widely adopted in inverse materials design to generate crystal structures with targeted compositions and properties. Although early implementations suggest scalability beyond traditional combinatorial methods, these models exhibit systematic limitations when applied to crystalline materials. This work identifies three interconnected failure modes that undermine their reliability. Mode collapse leads to overrepresentation of a narrow set of high-symmetry structures, neglecting structurally diverse candidates. Compositional violation results in chemically invalid outputs, including non-integer stoichiometries and charge imbalance. Stability loss arises when generated structures are thermodynamically or dynamically unstable, such as those lying above the convex hull or exhibiting imaginary phonon modes. These issues originate from a fundamental mismatch between diffusion models—designed for continuous, unconstrained data—and the discrete, periodic, and physically constrained nature of crystal systems. Based on peer-reviewed literature, this study provides a conceptual analysis of these failure modes and highlights that robust inverse design requires embedding physical constraints directly within the generative process rather than relying on post hoc filtering.