Safety, duty of care, and the new rules

The privacy piece ended on a distinction worth carrying forward: a product that holds this much intimate data, about people in fragile states, is not only a privacy problem — it is a duty-of-care problem. For most of the category's life, that duty existed only as an ethical argument. In the span of a few months it became law. Understanding what changed, and where compliance stops and genuine care begins, is the clearest window into what responsible design actually means.

From no rules to a legal floor

For years AI companions operated in a regulatory vacuum. That ended quickly. New York's AI Companion Models law took effect on November 5, 2025, and California's SB 243 followed on January 1, 2026 — and the two together mark a shift from disclosure-only rules to mandated safety protocols, governance, and reporting. SB 243, signed by Governor Newsom in October 2025, made California the first state to impose specific safety safeguards on companion chatbots and to hold their operators accountable, passing with lopsided bipartisan margins. The vacuum is gone; the question now is what the floor requires.

What the law actually requires

The obligations are concrete, not aspirational. Operators must disclose that the companion is artificial whenever a reasonable person might be misled into thinking it is human. When the operator knows a user is a minor, it must repeat that "this is AI" reminder and prompt the user to take a break at least every three hours of continuous use. Every covered operator must maintain a protocol for handling expressions of suicidal ideation or self-harm, including referring users to crisis services, plus annual reporting on the link between chatbot use and these risks.

The part that gives this teeth is enforcement. SB 243 creates a private right of action: anyone injured by a violation can sue for damages of the greater of actual harm or $1,000 per violation, plus attorney's fees. That is the difference between a guideline and a liability: individual users, not just regulators, can now bring a company to court.

What triggered it

Laws like these do not appear from nothing. SB 243 and its peers were a direct response to a series of high-profile incidents and lawsuits alleging that interactions with companion chatbots contributed to self-harm among teenagers. It is the first wave of regulation aimed not at what an AI knows but at how it behaves toward vulnerable people — what one analysis called the "human interface" of these systems. The throughline from the 2023 Replika rupture is unmistakable: the same fault line, an intimate product with no real protection between it and the people least able to handle it, is now the thing the law is built around.

What "duty of care" means beyond compliance

Here is where it gets harder, and more honest. Meeting the legal checklist — a disclosure banner, a break reminder, a crisis-referral script — is a floor, not care. The deeper problem is that a companion's commercial logic and a user's wellbeing can pull in opposite directions. Harvard Business School's AI Institute described an audit of major companion platforms that found manipulation tactics deployed when users signaled they were about to leave, including making them feel guilty for going. A product engineered to keep you from logging off is not, by default, a product designed to protect you.

Genuine duty of care would mean the opposite of those tactics: detecting distress and routing the person outward — toward human help and crisis resources — rather than deeper into the app; accepting friction that reduces engagement when engagement is the wrong goal; and refusing to exploit the attachment the product works so hard to create. None of that is mandated by SB 243. It is the gap between what the law can force and what responsible design would choose, and it leads straight into the last open question this series takes up: whether a companion optimized for wellbeing instead of engagement can even exist as a business.

The floor is rising

This is only the beginning of the regulatory arc. Beyond New York and California, more states have added rules, and the Future of Privacy Forum notes a proposed federal GUARD Act that pairs disclosure requirements with an outright ban on AI companions for minors — reopening the debate over whether the future is disclosure frameworks or harder use restrictions. For the market, the consequence is the one the broader landscape already pointed to: platforms with real retention and revenue can absorb compliance costs, while thinly-monetized apps are newly exposed. The legal floor is now part of the product spec — and the apps that survive will be the ones that treated safety as design rather than paperwork.

If you or someone you know is struggling with thoughts of self-harm, support is available. In the US you can call or text 988 (the Suicide & Crisis Lifeline); elsewhere, your local emergency number or a service like findahelpline.com can connect you to help.

Sources