"ChatGPT Health": Weaknesses in Medical Emergencies and Suicide Prevention

A recent study reveals significant vulnerabilities in ChatGPT Health's emergency response capabilities. The AI system provides potentially dangerous advice during medical emergencies and mental health crises, including suicide prevention situations. These findings highlight critical safety gaps in AI-driven healthcare tools, emphasizing the need for robust safeguards before deployment in high-risk medical scenarios. Human oversight remains essential for critical patient care.