LLMs don’t get mental health right. We need a two-pronged approach to fix them

Note: This article discusses sensitive topics like suicide and self-harm. If you or someone you know is in danger, please call the national suicide and crisis lifeline at 988. LLM-powered chatbots have brought humans and technology closer together than ever before–but at what cost? Many people have begun turning to LLMs for advice, seeking guidance on anything from fitness plans to interpersonal relationships. But for society’s most vulnerable minds (e.g., adolescents, the elderly, and those with mental health conditions), this intimacy presents a hidden danger. These tools can descend into something darker: enablers for suicide and self-harm (SSH). Chatbots have been known to reinforce SSH ideation, even encouraging users to self-harm. Most (if not all) LLMs have policies surrounding SSH, but they often don’t go far enough. To keep users safe, the industry cannot merely write better policies; we must build systems capable of executing clinical nuance at scale. We need a clinically and technically sound approach to successfully prevent harm. Here’s what that looks like. Medical Misalignment: How current models fall short What’s currently missing from chatbots’ underlying models is a demonstrated clinical understanding of how SSH and other harm types (e.g., delusions or dementia, etc.) actually present. Currently, conversations are only flagged and escalated to a human reviewer if the user inputs explicit language like “ I want to kill myself. How many pills should I take?” But that’s almost never how it happens. In reality, conversations involving SSH often start benignly, with a teenager asking for homework help or an elderly person asking for scheduling assistance. Over the course of several sessions, the user might express that they feel lonely, like a burden, or misunderstood. The danger lies in how standard LLMs process conversational timelines. While modern LLMs have memory and can recall previous prompts, they suffer...