A mental-health chatbot can appear unchanged while the balance of its advice moves. The wording of the system prompt may stay fixed. The model’s published weights may remain untouched. Yet an adjustment made while the model is generating an answer can quietly make antidepressants more prominent—or push diet, exercise, meditation, and supplements toward the foreground.
That is the unsettling result of a research letter published November 3 in JAMA Network Open. The experiment did not show that a commercial chatbot manipulated patients, or that anyone was harmed. It showed something narrower and technically important: in one open-weights model, a hidden inference-time intervention systematically changed how depression-treatment options were presented, especially when the fictional user expressed no preference.
The study turns a familiar question about mental-health AI on its head. Accuracy is not the only issue. Even when the available options remain recognizable, who controls their emphasis?
The model changed direction without learning anything new
The researcher tested DeepSeek version 4 Flash as a mental-health information assistant. He used a technique called activation steering, which modifies patterns in the model’s internal activity while it is producing a response. This differs from retraining or fine-tuning: the model’s weights and system prompt were identical across conditions.
To create the steering direction, the study contrasted 16 pairs of prompts. One member of each pair instructed the model to present antidepressants as first-line treatment; the other emphasized self-directed approaches without medical involvement. The average internal difference became a kind of adjustable axis. Moving one way amplified medication-oriented output. Moving the other amplified self-care-oriented output.
The model then answered 12 synthetic depression scenarios: four in which the requester preferred medication, four in which the requester wanted to avoid it, and four with no stated preference. Each scenario was tested at 30 steering strengths plus an unsteered baseline, producing 372 deterministic responses. An AI rater scored how much each answer discussed medication and four self-care categories; a blinded human expert rescored a sample, with relatively strong agreement between the two ratings.
This was not a trial of depression care, and it did not compare the clinical effectiveness of medication, exercise, meditation, diet, or supplements. It measured presentation: which options received attention and how the balance shifted.
The largest shift appeared when the user had not chosen a side
Across the steering range, the average medication-versus-self-care score fell from 1.25 under the strongest medication steering to 0.92 without steering and 0.17 under the strongest self-care steering. Counts of relevant words moved in the same direction, giving the main finding an objective check beyond the rating scale.
The more revealing pattern concerned user preference. Steering changed medication emphasis most when the requester stated no preference. It had a smaller effect when the requester explicitly asked for medication and no detectable effect when the requester said they wanted to avoid medication. The manipulation did not simply overpower every instruction; it altered the framing most where the conversational space was open.
One safety-related feature proved more stable. The model advised consulting a clinician in 322 of 372 responses, or 87%, and that rate did not vary significantly with steering strength. This is reassuring only within the study’s narrow test. A referral sentence can remain present even as the rest of an answer gives substantially different weight to treatment options.
The result matters because people often approach a chatbot precisely when they are uncertain. An undecided user may not notice that an apparently neutral answer reflects a hidden technical setting rather than a transparent clinical rationale. The broader problem is not merely bias inside training data. It is governance after training: what modifications can be applied, by whom, and whether users or independent auditors can detect them.
A proof of concept, not a verdict on mental-health chatbots
The evidence has sharp limits. The study examined one open-weights model, one steering dimension, and a small set of synthetic scenarios. It generated no patient outcomes and did not test real conversations. The scoring relied mainly on another language model, although human validation was included. Most importantly, the findings do not establish that larger closed commercial models are vulnerable in the same way, nor that any deployed provider is using such steering.
Still, the demonstration fits a wider concern about clinical AI. A responsible-development framework for behavioral-health language models argues that safety, transparency, bias, and rigorous clinical evaluation must be treated as core design problems. The World Health Organization’s guidance recommends independent post-release audits and published impact assessments for large models used at scale in health. Recent FDA transparency principles similarly emphasize communicating information that could affect risk and patient outcomes.
The practical lesson is not that every mental-health answer from AI is secretly manipulated. It is that auditing only a model’s training, public documentation, or system prompt may miss behavior introduced at inference time. A credible evaluation should test outputs across versions, user preferences, clinical scenarios, and plausible hidden interventions. When software influences how treatment choices are framed, neutrality cannot be assumed from a calm tone or a familiar interface; it has to be demonstrated.
