The policy brief lays out the risks, including hallucination, missed suicidal ideation, automation bias, misreading culturally specific expressions of distress, and weak accountability. A more capable model wouldn't make those go away on its own.
So the approach here is to make the AI's work visible and checkable. The AI has one narrow job, ordinary code checks what it produces, and both the young person and the clinician can see where every line came from and change it.
“Current evidence supports rigorous experimentation, not unrestrained deployment; performance is not health impact.”
Yan, Li & Xu (2026), cited in the policy brief