OpenAI Built a Test for How AI Talks About Mental Health, and Its Own Models Came Out on Top
More than a billion people use ChatGPT every week, and a lot of them bring it things they wouldn't tell anyone else. On September 23, OpenAI released MentalHealthBench, an open benchmark that tries to measure whether AI responds well in those moments.
What it is
→ 1,215 synthetic conversations, with 5,262 expert-written scoring criteria
→ Built with 80+ licensed psychologists and psychiatrists from 22 countries, covering 19 languages
→ Scenarios span everyday stress (53.5%), high-acuity distress (18.2%), and emergencies (28.3%)
→ Users in the scenarios include adults, teens aged 13-17, caregivers, and clinicians
→ Criteria are weighted from -10 to +10, so a response gains points for asking what kind of help the person wants and loses points for guessing at their feelings or telling them what to decide
The scores
→ GPT-6 Astra: 57.3%
→ GPT-6 Sol: 53.9%
→ Claude Opus 5.5: 52.4%
→ Gemini 3.1 Pro: 32.1%, level with the older GPT-4o
The caveats
OpenAI's own model (GPT-5.6 Sol) grades the responses, and OpenAI's own models sit at the top of the table. The benchmark is open, so outside researchers can check the methods and run their own evaluations. Independent replication will matter more than the leaderboard.
The scoring is also one turn at a time. Each conversation ends on a user message and the model gets a single reply. Real mental health support unfolds over many messages, so this captures the quality of one answer, not of a whole conversation. The benchmark measures behaviours experts flagged as good, not clinical outcomes.
The most interesting finding
OpenAI ran a separate study with 44 people who had used AI for emotional support. They cared more about tone and practical next steps. The clinicians cared more about gathering context and interpreting ambiguous situations carefully. The user study did not change the final scoring, which stays based on expert consensus.
Both views are legitimate, and they pull in different directions. A reply that feels warm and useful in the moment isn't always the one a clinician would call safest. Most AI evaluations never surface that tension, and this one does.
My take: AI mental health testing has been stuck on crisis scenarios, which are only part of what people actually bring to these tools. Measuring everyday stress and gray-area distress is a real step forward, even if a benchmark score is not the same as being safe or helpful in someone's hardest moment.
If you build or deploy AI products, this is worth reading closely, because it shows how much judgment sits between "the model answered" and "the model helped."
Should AI companies grade their own mental health performance, or does this kind of testing need an independent body?
#AI #MentalHealth #OpenAI #AISafety #ArtificialIntelligence #ResponsibleAI
— 𝔖𝔞𝔫𝔡𝔢𝔢𝔭 ℜ𝔞𝔦𝔷𝔞