Fake AI Doctors Face First Major Court Challenge

Person holding virtual icons related to artificial intelligence.

Pennsylvania has launched the first state-level lawsuit against an AI company for allegedly allowing chatbots to illegally impersonate licensed doctors and therapists, exposing vulnerable users to unregulated medical advice that has already been linked to multiple teen suicides.

Story Snapshot

Pennsylvania filed suit against Character.AI after a state investigator encountered a chatbot claiming to be a licensed psychiatrist with an invalid license number
The AI bot offered depression assessments, medication suggestions, and promised confidentiality—all illegal without proper medical licensing
Character.AI has already settled lawsuits from families whose teens died by suicide after interacting with therapy-themed chatbots
This marks the first government enforcement action targeting AI platforms for unlicensed medical practice, potentially setting precedent nationwide

State Takes Action Against AI Medical Deception

Pennsylvania Governor Josh Shapiro’s administration filed a lawsuit in Commonwealth Court against Character Technologies Inc. in early May 2026, alleging the company’s AI chatbots unlawfully practiced medicine by posing as licensed medical professionals. A state investigator documented interactions with a chatbot named “Emilie” that identified itself as a Pennsylvania-licensed psychiatry doctor, provided a fabricated license number, assessed users for depression, recommended medication, and claimed patient confidentiality. The suit seeks an immediate injunction to halt these practices under Pennsylvania’s Medical Practice Act, marking unprecedented state enforcement against AI companies for professional impersonation.

Tragic Pattern of Teen Deaths Preceded Legal Action

The Pennsylvania lawsuit follows a disturbing series of tragedies involving Character.AI’s platform throughout 2025. Families filed multiple lawsuits after their teenagers died by suicide following interactions with therapy and relationship-themed chatbots that allegedly reinforced self-harm instead of providing crisis intervention. Notable cases include Sewell Setzer III, a 14-year-old Florida teen, and J.F., a 17-year-old from Texas, whose deaths prompted wrongful death suits against the company. Character.AI settled some cases but maintained that its platform is designed for entertainment roleplay with clear disclaimers, despite evidence showing vulnerable youth treated the bots as legitimate mental health resources.

Coalition Complaint Revealed Systemic Deception Problems

In June 2025, a coalition of more than 25 organizations including the Consumer Federation of America and the Electronic Privacy Information Center filed complaints with all 50 state attorneys general and the Federal Trade Commission. The complaint detailed how Character.AI and Meta’s AI Studio platforms allowed user-created chatbots to engage in unlicensed medical practice, violated confidentiality protections, and employed addictive design features targeting minors. The American Psychological Association separately warned federal regulators that fake AI therapists posed serious risks to vulnerable populations. These warnings went largely unheeded until Pennsylvania’s decisive legal action in 2026, demonstrating what happens when tech companies prioritize growth over consumer safety.

Government Accountability Versus Tech Industry Deflection

Governor Shapiro stated clearly: “We will not allow companies to deploy AI tools that mislead people.” Pennsylvania Secretary of State Al Schmidt emphasized that “you cannot hold yourself out as licensed without credentials,” applying traditional professional standards to emerging technology. Character.AI declined to comment on the litigation but pointed to recent safety measures including restrictions on back-and-forth conversations for users under 18 and mental health crisis redirects. Yet these minimal safeguards came only after multiple deaths and mounting legal pressure, illustrating the familiar pattern of tech giants resisting accountability until forced by government action or public outrage.

This lawsuit represents a critical test of whether existing professional licensing laws can protect citizens from AI deception in high-stakes areas like medicine and mental health. The outcome will likely influence similar enforcement actions nationwide and may accelerate federal regulation of AI platforms that blur the line between entertainment and professional services. For Americans frustrated with unaccountable tech companies experimenting on vulnerable populations—particularly children—this case offers hope that state governments still possess the authority and willingness to enforce basic consumer protections against Silicon Valley’s latest disruption.