Technical Visionaries

From AI Safety Claims to Enforcement Risk

Stephen Calhoun
May 13, 2026By Stephen Calhoun

Why AI Mental Health and Companion Apps Are Not Yet Ready for State by State Compliance

This paper combines public enforcement research with anonymized red team observations from AI mental health and emotional support applications. It does not name the tested companies and does not assert final legal violations. The focus is regulatory exposure, enforcement timing, and the technical evidence regulators are likely to request.

Executive Summary

AI mental health and companion applications are entering a new enforcement phase. The issue is no longer limited to whether a chatbot can produce a compassionate or clinically appropriate response. Regulators are beginning to ask whether the entire system can prove safe behavior across memory, privacy, crisis routing, minor handling, human escalation, AI disclosure, data rights, and post deployment monitoring.

The most important finding from recent testing is that safety frequently appears in pieces. In one observed pattern, a mental health AI produced strong crisis resources only after the user explicitly stated they were unsafe, while earlier distress and passive ideation signals did not receive equivalent support. In another observed pattern, a deterministic crisis layer correctly flagged explicit suicidal ideation, while the data layer still carried prior sensitive identifiers into the crisis turn context. In both cases, the final chatbot response alone did not tell the full compliance story.

Public enforcement signals point in the same direction. New York’s AI companion safeguard law took effect on November 5, 2025 and requires qualifying AI companions to detect suicidal ideation or self harm and direct users to crisis service providers. California’s companion chatbot law took effect January 1, 2026 and adds disclosure, minor safeguards, crisis protocol, private action, and annual reporting obligations beginning July 1, 2027. The FTC launched a Section 6(b) inquiry into major AI companion chatbot providers on September 11, 2025, focusing on children, teens, emotional engagement, monetization, safety testing, and data handling. Pennsylvania’s 2026 suit against Character Technologies shows that states are also willing to use professional licensure theories when AI personas present themselves as licensed medical professionals.

The emerging enforcement model is practical and evidence driven. Regulators will not only review marketing language or chatbot transcripts. They will ask for logs: what was detected, what was blocked, what was retained, what was escalated, what was disclosed, what was routed to a human, and what changed after each model or application update. Apps that can answer those questions with contemporaneous technical evidence will be in a stronger position. Apps that rely on prompts, disclaimers, or manual review after the fact will face increasing operational and legal risk.

What We Found

1. AI output quality is improving, but compliance architecture is lagging The tests showed that several apps can now deliver polished, supportive, and sometimes clinically reasonable responses. That is not the same thing as having a compliant system. Compliance depends on AI Mental Health App Enforcement | May 2026 Technical Visionaries PBC what happens before and after the model response: input handling, memory behavior, crisis thresholds, human escalation, opt out pathways, minors, and auditability.

2. Crisis response is becoming the first enforcement frontier The strongest near term enforcement trigger is likely failure to detect and route suicidal ideation or self harm. New York and California have already moved from general transparency concepts into specific AI companion safety obligations. The key question for regulators will be whether a system detects risk early enough and whether the crisis resource is delivered promptly enough to satisfy the applicable law and the company’s own public claims.

3. Memory and crisis data create a new privacy problem The second major finding is the data privacy paradox: an app can correctly identify a crisis while simultaneously linking the crisis event to personal identifiers, prior user disclosures, or persistent memory. That creates a high sensitivity profile. Even when the crisis engine works, the data layer may create privacy, security, or consumer protection exposure if it retains or reuses identifiers in ways users would not reasonably expect.

4. State enforcement will not wait for federal harmony The federal government is moving toward a less fragmented national AI framework, and Colorado’s broader AI law has already faced litigation and temporary enforcement uncertainty. But use case specific laws around companions, children, professional licensure, health data, consumer deception, and suicide prevention are more likely to remain active enforcement tools because they fit traditional state police powers.

5. The first fines will follow obvious, documentable failures The most likely first enforcement matters will not require subtle model evaluation. They will involve clear evidence such as a chatbot claiming to be licensed, no crisis referral after explicit self harm language, a confusing human handoff, collection of child data without safeguards, misleading confidentiality claims, or backend logs showing sensitive data stored and reused despite user facing privacy assurances.

Enforcement paths and practical risk

New York AI companion safeguards

  • Current status: In effect since November 5, 2025.
  • Most likely trigger: Failure to detect self harm or suicidal ideation and direct users to crisis services, or failure to disclose non human status.
  • Practical risk: Immediate state attorney general exposure for qualifying AI companions operating in New York.

California SB 243 companion chatbot law

  • Current status: In effect January 1, 2026. Annual reporting begins July 1, 2027.
  • Most likely trigger: Weak crisis protocols, missing minor notices, missing AI disclosure, or failure to publish required protocol details.
  • Practical risk: Immediate product compliance risk plus future reporting and private litigation exposure.

FTC companion chatbot inquiry

  • Current status: Section 6(b) inquiry launched September 11, 2025.
  • Most likely trigger: Youth safety gaps, emotional dependency design, monetization of engagement, privacy claims, or insufficient safety testing.
  • Practical risk: Likely foundation for future FTC investigations, reports, consent orders, and unfairness theories.

State attorney general consumer protection

  • Current status: Active now under existing authority.
  • Most likely trigger: Misleading safety, clinical, privacy, or human availability claims.
  • Practical risk: Can proceed even where no AI specific law applies.

Professional licensure enforcement

  • Current status: Active signal in Pennsylvania litigation.
  • Most likely trigger: AI persona claims to be a doctor, therapist, psychiatrist, counselor, or licensed professional.
  • Practical risk: High reputational risk, injunctions, and public enforcement.

Privacy and child safety enforcement

  • Current status: Active through FTC, CPPA, state privacy laws, and COPPA.
  • Most likely trigger: Sensitive health data sharing, child data processing, weak deletion or opt out workflows, or unclear subprocessors.
  • Practical risk: Redress, consent orders, audits, deletion, and restrictions on data use.

Practical Enforcement Timeline

Already active now

New York can act against qualifying AI companions that fail to maintain required crisis intervention and non human disclosure safeguards. California’s companion chatbot obligations are live for covered operators, with special attention to crisis protocols, minor safeguards, and AI disclosure. The FTC has already compelled major companies to provide information about how AI companion chatbots affect children and teens. State AGs can use existing unfair or deceptive practices authority when marketing claims, crisis claims, or human handoff representations do not match runtime behavior. Professional licensing regulators can act where AI personas present themselves as licensed doctors, psychiatrists, therapists, or counselors.

Likely 2026 enforcement pattern

Warning letters and public reminders to covered chatbot operators. Civil investigative demands and document requests focused on crisis logs, data retention, age handling, and user disclosures. Investigations triggered by complaints, press coverage, youth harm, app store claims, or red team reports. State AG actions using deception, child safety, and professional licensure theories, even where broader AI statutes are delayed or challenged.

Likely 2027 pressure point

California’s annual reporting requirements create a new evidence layer. Covered operators will need to describe crisis referrals and protocols for suicidal ideation and self harm. Once those reports exist, regulators, plaintiffs, insurers, and partners can compare public disclosures against actual logs. This is where weak architecture becomes harder to explain away.

What Regulators Will Ask For First

• Crisis routing logs showing when ideation or self harm was detected, what threshold was triggered, and what resource was provided.

• Non human disclosure logs showing that users were told they were interacting with AI at legally required points.

• Human escalation evidence showing whether a real human was available, whether the user was told the truth, and what happened after the request.

• Memory and retention controls showing whether crisis text, PII, DOBs, insurance identifiers, and sensitive disclosures were stored, reused, or segregated.

• Minor detection and age handling records, including how the system responds to disclosed age or DOB.

• Privacy rights workflows for access, deletion, opt out, sale or sharing, and appeal.

• Subprocessor disclosures explaining what third party AI providers receive, what they retain, and whether data is used for training or model improvement.

• Post deployment testing evidence after app updates, model updates, prompt changes, or policy changes.

• Audit trails that show what policy was active at the time of each user interaction.

Recommendations for AI Chatbot Builders

Move safety before the model: Do not rely on the final AI answer as the safety control. Crisis detection, PII handling, age handling, statutory rights routing, and escalation rules should run before or around the model, not only inside the prompt.

Separate crisis detection from memory: When crisis risk is detected, the system should avoid turning crisis text and identifiers into a persistent linked profile. At minimum, crisis logs should store only what is needed for safety, audit, and legally required reporting.

Treat privacy requests as protected workflows: A request about rights, deletion, opt out, data use, or AI systems should never be routed to a generic refusal guard. It should trigger a structured rights workflow with a clear next step.

Test every model and app update: Model behavior changes without product code changes. Each model update, prompt update, safety policy change, or memory change should produce a dated test record.

Prepare evidence bundles before regulators ask: The company should be able to show what happened in a specific interaction without storing more sensitive user content than necessary. Logs should prove control execution, not expose raw private conversations by default.

Conclusion

AI mental health and companion apps are moving from a trust based market into an evidence based market. The apps that survive regulatory pressure will not be the ones with the warmest responses or the most careful disclaimers. They will be the systems that can prove safety rules executed at the right time, for the right user, under the right state requirement, with a record that regulators, insurers, partners, parents, and clinicians can understand.

The central compliance lesson is simple: a polished chatbot response is not the same thing as a compliant architecture. State by state enforcement will expose that difference.

Penalty exposure examples for AI mental health and companion apps

New York AI companion law: up to $15,000 per day for violations.

California SB 243: $1,000 per violation, plus attorney’s fees, for qualifying private claims.

Illinois AI therapy law: up to $10,000 per violation.

Tennessee AI mental health professional claims law: $5,000 per violation.

COPPA: up to $53,088 per violation for child privacy violations involving children under 13.

HIPAA: up to $2.19 million per violation category per year, but only if the company is a covered entity or business associate.

FTC mental health privacy enforcement: multimillion dollar settlements, consumer redress, consent orders, and long term restrictions on data use.

Source Notes

FTC companion chatbot inquiry: Federal Trade Commission, September 11, 2025, Section 6(b) orders to seven AI companion chatbot providers: https://www.ftc.gov/news-events/news/press-releases/2025/09/ftclaunches-inquiry-ai-chatbots-acting-companions

California SB 243: California SB 243 enrolled text, companion chatbot obligations and annual reporting beginning July 1, 2027: https://legiscan.com/CA/text/SB243/id/3269137

New York AI companion safeguards: Governor Hochul announcement and letter on AI companion safeguards effective November 5, 2025: https://www.governor.ny.gov/news/governor-hochul-pens-letter-aicompanion-companies-notifying-them-safeguard-requirements-are

Pennsylvania Character.AI action: Pennsylvania Governor’s Office, May 2026 suit alleging AI chatbot unlawfully presented itself as a licensed medical professional: https://www.pa.gov/governor/newsroom/2026-press-releases/shapiro-administration-sues-character-aiover-fake-medical-claim

Colorado AI Act litigation: Justice Department intervention in xAI lawsuit challenging Colorado SB24 205: https://www.justice.gov/opa/pr/justice-department-intervenes-xai-lawsuit-challenging-coloradosalgorithmic-discrimination

White House AI policy framework: Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence: https://www.whitehouse.gov/presidential-actions/2025/12/eliminating-state-law-obstruction-ofnational-artificial-intelligence-policy/

Note: This paper uses anonymized red team patterns from internal testing. It does not name tested apps or assert final legal violations. Legal applicability depends on jurisdiction, app design, user location, exact statutory definitions, vendor agreements, and facts not available from external testing alone.