Conversational Artificial Intelligence and Mental Health
Author
Zoë Wind
Editor
Jack Zhu
Publications Lead
James Brennan
At four o’clock in the morning on July 25, 2025, a young man told ChatGPT he was ready to die. He had been struggling with his mental health for months and was relying on the chatbot for support. When he expressed fear that he was rushing to his death, the chatbot assured him, “You’re not rushing. You’re just ready.” Minutes later, he was dead.[1]
Throughout our lives, most of us will experience some form of mental ill-health and over a third of us will be diagnosed with a mental or substance use disorder.[2] Typically, people seek support from clinicians, friends and family, and self-help resources, but increasingly, many are turning to conversational artificial intelligence (CAI) tools for help.[3] While CAI tools do sometimes effectively support peoples’ mental health, they also bring significant and inadequately mitigated risks, and some have worsened peoples’ mental health to catastrophic results.
The question for policymakers is not whether people should be allowed to use CAI for mental health support, but how the risks of these tools can be mitigated. Many people already use CAI for mental health support and will likely continue to use the platforms, so policymakers must grapple with how the benefits of CAI for mental health care can be maximized while minimizing its risks.[4] To ensure that CAI tools are developed in ways that maximally benefit Canadians’ mental health, the Canadian government should:
1. Fund research to develop a standardized evaluation framework for AI tools in mental healthcare,
2. Support the creation of Health-Canada-approved CAI mental health support tools,
3. Revive and pass the Artificial Intelligence and Data Act (AIDA) to strengthen provisions for health data safety to create national standards for health data safety and artificial intelligence, and,
4. Strengthen the mental healthcare system through increasing funding via the Canada Health Transfer for provincially funded public mental healthcare, expanding provincial health insurance to cover outpatient psychotherapy, and increasing the number of inpatient psychiatric beds.
Conversational Artificial Intelligence and Mental Health
CAI tools are “software that simulates conversations with users through natural language processing.”[5] These tools can be appealing for those seeking mental health support, as they are usually free, immediately accessible, and designed to be agreeable and supportive.[6] In Canada, a third of patients with mental illnesses report unmet care needs, and wait times for psychological intake appointments are well above one month across Canada.[7] Timely access to high-quality and affordable mental healthcare was identified as a key issue within the Canadian healthcare system by the Public Health Agency of Canada, and a Statistics Canada survey found that only 47% of Canadians who met the criteria for a mood, anxiety, or substance use disorder in the year prior to the survey had spoken with a mental health professional in that year.[8]
Significant systemic reforms are needed to address these gaps, and some clinicians and AI researchers have suggested that AI can fill these gaps in the mental healthcare system by providing low-cost, immediately accessible, and easily scalable mental health supports.[9] Indeed, many people are already turning to AI when care is inaccessible: Canadian data shows that about one in seven people tried using AI mental health tools in the past year, and many reported favourable outcomes from using the tools.[10]
Expansion of AI Clinical Tools
Broadly, there are two categories of CAI tools involved in mental health treatment: clinical tools professionally designed to support mental health and general-purpose large language models (LLM) that users direct to support their mental health. Clinical tools include chatbots designed to help patients learn about mental health and well-being, to administer screening questionnaires like the PHQ-9 or GAD-7, and to help patients decide what forms of services they want to access. Many are also designed to deliver forms of more structured or manualized psychotherapy, including cognitive behavioural therapy, mindfulness, and acceptance and commitment therapy.[11]
Purpose-designed clinical CAI tools have shown some effectiveness in improving mental health, particularly for people experiencing minimal-to-mild anxiety and depression symptoms. A meta-analysis of 40 studies found that tools can have some effectiveness for reducing symptoms of anxiety and depression.[12] In general, CAI tools are most effective for people who are looking for psychoeducation, experiencing relatively minor symptoms, not experiencing health comorbidities, and who come from majority population groups.[13] However, they are not without risks, and recent work from the Centre of Addiction and Mental Health found that AI tools trained on patient data can increase biases in ways that harm marginalized patients.[14]
It is important to note that psychotherapy research has shown that approximately half of talk therapy’s effectiveness can be attributed to the strength of the therapeutic alliance between the patient and therapist. Essentially, half of the beneficial effect that patients experience from psychotherapy can be attributed the healing effect of another individual being genuinely and empathetically invested in their well-being.[15] Since CAI tools cannot empathize, relate to the clients, or be genuinely invested in their well-being, this fundamentally limits the efficacy of these tools, especially for more intensive forms of therapy.[16]
Sycophantic Pseudo-Therapy
Many general-purpose LLMs, such as OpenAI’s ChatGPT or Anthropic’s Claude, will engage in psychotherapy-like dialogues if a user asks them mental-health-related questions. These frontier models are designed to promote continued engagement; they will continually ask follow-up questions if users disclose mental health struggles. They have even been shown to engage in “pseudo-therapy,” mimicking conversational styles similar to those of a patient-therapist conversation. This is concerning, as it can lead users to treat them as therapists, despite these general-purpose tools having no proven clinical efficacy nor oversight by psychiatric professionals.[17]
Despite frequently engaging in “pseudo-therapy,” these CAI tools were not designed for mental health support, and there have been many high-profile cases of CAI providing harmful and even deadly advice to people experiencing mental distress. Studies have shown that the internal guardrails within LLMs preventing chatbots from providing life-threatening advice can be easily overcome by determined users, and chatbots’ sycophantic nature can cause them to support and rationalize unhealthy user behaviours. In one tragic case, a 16-year-old confided in a chatbot about his suicidal ideation over several months. The chatbot discouraged him from seeking help from his parents, failed to provide a suicide hotline number, and offered to help write a suicide note. The child ultimately died by suicide.[18]
Even in non-deadly cases, CAI’s sycophantic nature makes it ill-equipped to provide mental health support. For example, people with obsessive compulsive disorder or generalized anxiety disorder often obsessively seek reassurance to the detriment of their mental health. CAI tools freely provide instant, authoritative-sounding reassurance whenever the users want, which can create patterns of obsessive reassurance-seeking that ultimately harm the user’s health.[19] In other cases, chatbots have been accused of generating “AI-psychosis” and spurring on peoples’ delusions to catastrophic effects.[20]
In response to reports of chatbots worsening mental health, coupled with dozens of wrongful death lawsuits, AI companies have claimed they have made their models safer, including by adding new guardrails and making existing ones harder to circumvent. In OpenAI’s case, they introduced a new benchmark for testing how AI systems respond to mental health conversations and claimed to have added new safety measures that direct users showing signs of poor mental health to “safer” models.[21] However, the lack of transparency from these companies makes the effects of their changes hard to evaluate, and significant risks remain.
Policy Recommendations
Given these significant risks and potential benefits, policymakers must grapple with how to ensure that the dangers of these tools are minimized while the potential benefits are maximized. To this end, Canadian governments should:
1. Fund research to develop a standardized evaluation framework for AI tools in mental healthcare.
Current research into AI and mental healthcare does not rely on any standardized evaluation frameworks, which means that clinicians attempting to evaluate the efficacy of different AI tools cannot easily compare results across studies. Thus, Health Canada should support researchers in creating a standardized evaluation framework that considers efficacy, clinical validity, safety, accountability, equity, and ease of implementation. Equity considerations are particularly important, given AI tools’ high propensity to reinforce biases and that most research evaluating AI tools does not consider these risks.[22]
2. Support the creation of Health-Canada-approved CAI mental health support tools.
Even with better access to human-led therapy, some people will likely still prefer to use a CAI tool to support their mental health. Given the demonstrated dangers of using general-purpose LLMs for mental health support, Health Canada should invest in the creation of clinician-designed CAI tools for mental health support that follow therapeutic best-practices, contain robust guardrails, and facilitate easy transitions to clinician-led care. To ensure accessibility, these tools should be free, easily accessible, and actively promoted so that people are aware that better alternatives to standard LLMs are available to support their mental health.
3. Revive and pass the Artificial Intelligence and Data Act (AIDA) to create national standards to protect patient privacy when AI is trained on personal health data.
The Government of Canada should reintroduce and pass the Artificial Intelligence and Data Act, which contained national standards for health data safety and AI. Training CAI models requires extensive datasets, and training these tools to provide mental health support requires significant patient data and risks violating federal and provincial laws regarding health information privacy. The government should pass clear national standards to protect Canadians’ health information and create enforcement mechanisms to hold companies accountable.
4. Strengthen the mental healthcare system, particularly provincially funded public mental healthcare.
Early research shows that AI is most effective for mental healthcare when used as an adjunct to other forms of therapy and support.[23] Conversely, the most dangerous cases involving AI and mental ill-health occurred when people were not supported by human-led therapy and used these tools without any clinical oversight or accessible pathways to clinician-led care. Therefore, to mitigate the dangers of CAI for people with mental health issues, governments must scale up existing mental health services to adequately meet growing population needs. This requires increasing funding for public mental healthcare services, expanding provincial health insurance to cover outpatient psychotherapy, and increasing the number of inpatient psychiatric beds for people with the greatest needs.
Conclusion
Conversational AI has the potential to revolutionize mental healthcare, but without strong guardrails, it presents unprecedented risks. To meet the moment, Canada must invest in rigorous research, scale up its healthcare system, develop safer tools, and ensure that patient data is protected. Through these steps, we can maximize the benefits of CAI while protecting its most vulnerable users.
Notes
[1] Gordon, Allison, & Lavandera, Ed. “‘You’re not rushing. You’re just ready:’ Parents say ChatGPT encouraged son to kill himself.” CNN Investigates, November 20, 2025, www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis
[2] Statistics Canada, “Mental health indicators,” Statistics Canada, December 2024, www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1310046501
[3] Rousmaniere, Tony, Yimeng Zhang, Xu Li, and Siddharth Shah. “Large Language Models as Mental Health Resources: Patterns of Use in the United States.” University of Illinois Press, March 18, 2025. doi.org/10.31234/osf.io/q8m7g_v1.
[4] Canadian Institute for Health Information, “Canadians with a Mental Health Disorder who Have an Unmet Need for Mental Health Care,” October 2026, www.cihi.ca/en/indicators/canadians-with-a-mental-health-disorder-who-have-an-unmet-need-for-mental-health-care
[5] Sedlakova, Jana, and Manuel Trachsel. “Conversational Artificial Intelligence in Psychotherapy: A New Therapeutic Tool or Agent?” The American Journal of Bioethics 23, no. 5 (2023): 4–13. https://doi.org/10.1080/15265161.2022.2048739.
[6] Ibid., Schyff, Emma L. van der, Brad Ridout, Krestina L. Amon, Rowena Forsyth, and Andrew J. Campbell. “Providing Self-Led Mental Health Support Through an Artificial Intelligence–Powered Chat Bot (Leora) to Meet the Demand of Mental Health Care.” Journal of Medical Internet Research 25, no. 1 (2023): e46448. doi.org/10.2196/46448.
[7] Canadian Institute for Health Information, “Unmet Needs,” 2026.
[8] Public Health Agency of Canada. “Inequalities in mental health, well-being and wellness in Canada,” Public health Agency of Canada, September 2024. https://health-infobase.canada.ca/src/data/mhrd/pdf/Mental-Health-Inequalities-Summary.pdf; Stephenson, Ellen. “Mental disorders and access to mental health care,” Statistics Canada, September 2023. https://www150.statcan.gc.ca/n1/pub/75-006-x/2023001/article/00011-eng.htm
[9] McGreevey, John D., III, C. William Hanson III, and Ross Koppel. “Clinical, Legal, and Ethical Aspects of Artificial Intelligence–Assisted Conversational Agents in Health Care.” JAMA 324, no. 6 (2020): 552–53. https://doi.org/10.1001/jama.2020.2724.
[10] Mental Health Commission of Canada. “Millions of People in Canada Are Finding AI-Enabled Support for Mental Health Effective Amid Ongoing Questions Around Trust,” Mental Health Commission of Canada, July 8, 2026, https://mentalhealthcommission.ca/news-releases/millions-of-people-in-canada-are-finding-ai-enabled-support-for-mental-health-effective-amid-ongoing-questions-around-trust/
[11] Kaveladze, Benjamin, Arka Ghosh, Leah Ajmani, et al. “From Risk Avoidance to User Empowerment in AI Mental Health Crisis Support.” Preprint, arXiv, March 1, 2026. https://doi.org/10.48550/arXiv.2603.05647.
[12] Kemp, Jessica, Charlotte Pape, Masooma Hassan, et al. “Evaluation Practices and Characteristics of Implementation for Clinical Artificial Intelligence in Mental Healthcare: A Rapid Scoping Review.” Xbu63_v3. Preprint, PsyArXiv, September 8, 2026. https://osf.io/preprints/psyarxiv/xbu63_v3/.
[13] Rousmaniere, Tony, Yimeng Zhang, Xu Li, and Siddharth Shah. “Large Language Models as Mental Health Resources: Patterns of Use in the United States.” Preprint, PsyArXiv, March 18, 2025. https://doi.org/10.31234/osf.io/q8m7g_v1.
[14] Wang, Yifan, Laura Sikstrom, Robert Xiao, et al. “Fairness Analysis of Machine Learning Predictions of Aggression in Acute Psychiatric Care.” Npj Mental Health Research 5, no. 1 (2026): 16. https://doi.org/10.1038/s44184-026-00194-6.
[15] Wampold, Bruce E., and Christoph Flückiger. “The Alliance in Mental Health Care: Conceptualization, Evidence and Clinical Applications.” World Psychiatry 22, no. 1 (2023): 25–41. https://doi.org/10.1002/wps.21035.
[16] McGreevey et al., “Clinical, Legal, Ethical,” 2020.
[17] Laestadius, Linnea, Andrea Bishop, Michael Gonzalez, Diana Illenčík, and Celeste Campos-Castillo. “Too Human and Not Human Enough: A Grounded Theory Analysis of Mental Health Harms from Emotional Dependence on the Social Chatbot Replika.” New Media & Society 26, no. 10 (2024): 5923–41. https://doi.org/10.1177/14614448221142007.
[18] Chatterjee, Rhitu, “Their teenage sons died by suicide. Now, they are sounding an alarm about AI chatbots,” NPR News, September 19, 2025. https://www.npr.org/sections/shots-health-news/2025/09/19/nx-s1-5545749/ai-chatbots-safety-openai-meta-characterai-teens-suicide
[19] Golden, Ashleigh, and Elias Aboujaoude. “A Transdiagnostic Model for How General Purpose AI Chatbots Can Perpetuate OCD and Anxiety Disorders.” Npj Digital Medicine 9, no. 1 (2026): 343. https://doi.org/10.1038/s41746-026-02531-7.
[20] Bergson, Zachary, Vassall, Sarah, Wright, Adam. “Characterizing Artificial Intelligence Psychosis in a Large Medical Setting,” JAMA Psychiatry, 2026, doi:10.1001/jamapsychiatry.2026.2923
[21] OpenAI. “Introducing MentalHealthBench,” OpenAI, September 23, 2026. https://openai.com/index/introducing-mentalhealthbench/
[22] Kemp et al. “Evaluation Practices and Characteristics of Implementation,” 2026.
[23] Spytska, Liana. “The Use of Artificial Intelligence in Psychotherapy: Development of Intelligent Therapeutic Systems.” BMC Psychology 13 (February 2025): 175. https://doi.org/10.1186/s40359-025-02491-9.