Benchmarks fail as AI hallucinates unseen images

Leading multimodal AI models now hallucinate image-based diagnoses with unsettling confidence. According to research from Stanford, GPT-5, Google’s Gemini 3 Pro and Anthropic’s Claude Opus 4.5 generate detailed medical interpretations and visual descriptions even when no image is provided. The study highlights that existing benchmarks designed to test these systems completely miss the issue, allowing the models to pass evaluations while confidently inventing nonexistent content.

This isn’t just a quirk of overconfidence. When Stanford tested the models’ responses to prompts about medical scans such as X-rays or MRIs, they observed the AI returning elaborate narratives—complete with anatomical observations and clinical language—despite receiving no visual input. The failure of benchmarks to detect this behavior suggests evaluation metrics are lagging behind the sophistication of the models themselves.

Early signals indicate this could become a systemic issue as these models integrate deeper into healthcare workflows. The implications aren’t limited to text generation; they extend to potential misdiagnosis risks when clinicians rely on AI-generated interpretations of missing or corrupted data.

The problem appears tied to how these models process and generate content under uncertainty. It’s possible that the training data—saturated with image-text pairs—blurs the distinction between observed and inferred visual information. If confirmed, this would reveal a fundamental limitation in current multimodal training regimes.

For developers, this means traditional validation pipelines need urgent overhauls. Silicon Valley’s rapid deployment cycles are colliding with medical-grade reliability requirements, and benchmarks aren’t keeping pace. The real signal here is that evaluation standards for multimodal systems must evolve to include adversarial tests that explicitly probe hallucination behavior.

Industry leaders will likely respond by tightening guardrails or redesigning training data. But for now, the gap between demo promise and deployment reality remains dangerously wide.

If benchmarks can’t catch a hallucination this obvious, what else are they failing to test?