Anthropic says the conversations make Claude show ‘apparent distress.’

On Friday, Anthropic said its Claude chatbot can now end potentially harmful conversations, which "is intended for use in rare, extreme cases of persistently harmful or abusive user interactions." In a press release, Anthropic cited examples such as sexual content involving minors, violence, and even "acts of terror."

Anthropic has introduced a new feature in its Claude Opus 4 and 4.1 models that allows the generative AI (genAI) tool to end a conversation on its own if a user repeatedly tries to push harmful or illegal content.