Harmful, abusive interactions plague AI chatbots. Researchers have found that AI companions like Character.AI, Nomi, and Replika are unsafe for teens under 18, ChatGPT has the potential to reinforce users’ delusional thinking,

The Claude AI models Opus 4 and 4.1 will only end harmful conversations in “rare, extreme cases of persistently harmful or abusive user interactions.”

Anthropic has given its chatbot, Claude, the ability to end conversations it deems harmful. You likely won't encounter the feature, however.