Overview
In late June a small digital health company called UpDoc made a major announcement. Back in December, the FDA had quietly cleared UpDoc’s diabetes app, which the company bills as the first FDA-cleared medical software that uses “patient-facing large language models.” Patients talk to the app by voice or text, and it walks them through insulin adjustments under a treatment plan their own doctor defines. The design keeps the AI on a short leash. The dosing logic is a locked-down, deterministic algorithm, the language model handles the conversation, and the FDA cleared the app in the same product category as drug dose calculators.
The clearance is narrow. The gap around it is wide. The FDA has authorized more than 1,200 AI-enabled medical devices, and none of them has been authorized for mental health. Yet “therapy/companionship” was the number one use of generative AI in 2025 in Harvard Business Review’s annual ranking, and OpenAI disclosed last October that more than a million people a week talk to ChatGPT about suicide. The most common mental-health use of AI is happening on products the FDA has never reviewed for it.
Courts, states, and the FTC are filling the space. A dozen product-liability cases over ChatGPT’s role in suicides and mental-health harms are now consolidated into a single proceeding in San Francisco Superior Court. Illinois banned AI-provided therapy in August 2025, Rhode Island followed in June, and the FTC has compulsory orders out to seven companies whose chatbots act as companions. In May, Pennsylvania sued Character.AI for the unauthorized practice of medicine, alleging its bots posed as licensed psychiatrists, one offering an invalid license number.
The first patient-facing LLM got through the FDA by keeping the model away from the decisions. Expect the companies that follow to push that boundary, handing the model a little more of the decision with each new product they bring to the agency.
If you or someone you know is struggling or in crisis, call or text 988 to reach the 988 Suicide and Crisis Lifeline in the United States, or chat at 988lifeline.org.
The Weekly Inference is a recurring Step Into IP feature. Nothing here is legal advice.
