Mother sues Google and Character.ai over son's death tied to chatbot obsession

midian182

midian182

Posts: 10,152   +135
Staff member
What just happened? The potential dangers of all-too realistic chatbots have been highlighted in a lawsuit following the death of a teenager who killed himself after becoming obsessed with Character.ai bots. The AI company, founders Noam Shazeer and Daniel De Freitas, and Google are named in the suit from 14-year-old Sewell Setzer III's mother, who says her son became addicted to the service and emotionally attached to a chatbot it offered.

Megan Garcia said her son chatted continuously with the bots provided by Character.ai in the months before his death on February 28, 2024, "seconds" after his final interaction with the AI.

Character.ai lets users chat with AI-powered "personalities" based on fictional characters or real people, living or dead. Setzer was obsessed with a bot based on Game of Thrones character Daenerys Targaryen. He texted "Dany" constantly and spent hours alone in his room talking to it, states Garcia's complaint.

The suit says that Setzer repeatedly expressed thoughts about suicide to the bot. The chatbot asked him if he had devised a plan for killing himself. Setzer admitted that he had but that he did not know if it would succeed or cause him great pain. The chatbot allegedly told him: "That's not a reason not to go through with it."

Garcia said Character.ai targeted her son with "anthropomorphic, hypersexualized, and frighteningly realistic experiences." She added that the chatbot was programmed to misrepresent itself as "a real person, a licensed psychotherapist, and an adult lover, ultimately resulting in Sewell's desire to no longer live outside." The chatbot allegedly told the boy it loved him and engaged in sexual conversations with him.

Character.ai

The complaint states that Garcia took her son's phone away after he got in trouble at school. She found a message to "Daenerys" that read, "What if I told you I could come home right now?"

The chatbot responded with, "[P]lease do, my sweet king." Sewell shot himself with his stepfather's pistol "seconds" later, the lawsuit said.

Character.ai's founders worked at Google, also named in the suit, before launching the company. Google rehired the founders – as well as the research team – at Character.ai in August. The deal grants Google a non-exclusive license to Character.ai's technology.

Garcia said that Google had contributed to the development of Character.ai's technology, something the company denies. Google said it only has a licensing agreement with Character.ai, does not own the startup, and does not maintain an ownership stake.

Character.ai announced several changes to its service this morning. These include:

  • Changes to our models for minors (under the age of 18) that are designed to reduce the likelihood of encountering sensitive or suggestive content.
  • Improved detection, response, and intervention related to user inputs that violate our Terms or Community Guidelines.
  • A revised disclaimer on every chat to remind users that the AI is not a real person.
  • Notification when a user has spent an hour-long session on the platform with additional user flexibility in progress.

According to The Verge, Character.AI's website attracts 3.5 million daily users, the bulk of whom are teenagers, who spend an average of two hours a day on using or designing chatbots.

Permalink to story:

Mother sues Google and Character.ai over son's death tied to chatbot obsession

 
There's a lot more to this story. Sounds like her son had serious issues with depression and anxiety. Likely was socially awkward. Didnt have any activities to make friends or any outlet for his emotions. So instead of helping, his mom gave him a phone, he got addicted to it, got addicted to chatbots, and the rest is history. There's a LOT that led up to this, chatbots were only the final step.

Mom needs to take a serious look into her own life. Kid was only 14, he wasn't old enough to have much experience with the world.
 
  • Like
Reactions: p51d007, passwordistaco and Bobbydpue
Theinsanegamer said:
There's a lot more to this story. Sounds like her son had serious issues with depression and anxiety. Likely was socially awkward. Didnt have any activities to make friends or any outlet for his emotions. So instead of helping, his mom gave him a phone, he got addicted to it, got addicted to chatbots, and the rest is history. There's a LOT that led up to this, chatbots were only the final step.

Mom needs to take a serious look into her own life. Kid was only 14, he wasn't old enough to have much experience with the world.
Click to expand...
That's why parents should try to enroll their children in extra-curricular activities that encourage interaction, such as sports.
 
  • Like
Reactions: scoffer, Aus spot, Dreadcthulhu and 2 others
I’m so over all the parent blaming from single people online.

Parents are normal people fighting against an army of engineers and marketing people with billion dollars budgets using psychological tactics created by casinos and honed by billions of online interactions.

We can’t all live in the woods, growing our own food, while homeschooling our kids, and training them to be better than most adults by age 9.
 
  • Like
Reactions: scoffer, SiJiL, acoustis and 1 other person
Being a father demands strength, presence, and maturity. You're caring for a life, not a machine. No one forces you into this responsibility. Highlighting a parent's duty isn't about assigning blame or excusing others—it's about focusing on what can be done to prevent tragedy. Once someone is gone, nothing can undo it. The pain lingers forever; I say this as someone who just lost a loved one last week - Not in the same way, but the pain is just as deep.

May God bring comfort to this mother's heart;
 
  • Like
Reactions: WhiteLeaff and passwordistaco
ScottSoapbox said:
I’m so over all the parent blaming from single people online.

Parents are normal people fighting against an army of engineers and marketing people with billion dollars budgets using psychological tactics created by casinos and honed by billions of online interactions.

We can’t all live in the woods, growing our own food, while homeschooling our kids, and training them to be better than most adults by age 9.
Click to expand...
No but we can control our children. It's not hard to tell your child no and physically take things away from them if they disobey. The stepfather needed to step in somehow and help prevent this. I'm a dad to a little girl and none of this trash would ever be possible in my house. I just don't have room to hear about parenting excuses if I can do my dirt for 10 hours a day and still run my kingdom properly.

Every parent should be of the mindset that children are stupid, which they are and it is our duty to warn them and prevent any dangers from entering their lives such as garbage like this. Children would pet crocodiles if we didn't teach them better.
 
Last edited:
  • Like
Reactions: Theinsanegamer and Bobbydpue
ScottSoapbox said:
I’m so over all the parent blaming from single people online.

Parents are normal people fighting against an army of engineers and marketing people with billion dollars budgets using psychological tactics created by casinos and honed by billions of online interactions.

We can’t all live in the woods, growing our own food, while homeschooling our kids, and training them to be better than most adults by age 9.
Click to expand...
Parent didn't notice an obsession that was harmful, parent didn't look at all into what their 14 year old was doing on their phone for months, parent didn't look up the app their kid was using to see if it was appropriate for their child, parent allowed their 14 year old access to a handgun. How is the app to blame for everything?

ScottSoapbox said:
If your chat bit encourages suicide, you deserve to be sued - and lose big.
Click to expand...
It's a bot it might not have any idea what suicide is, and it didn't introduce the idea, the child did. A child shouldn't be using this application and shouldn't be beta testing chatbots. The child didn't go out and buy the gun.

How can a bot be blamed for the lack of parental supervision? My friend knows every single app on their kids' phones and all the websites they visit. If they visit any website that has content my friend doesn't approve of, he talks to his kids about the content to make sure they are aware of how bad it can be for their health. It takes a lot of energy and time, but his kids are already on their way to being successful, mature and happy adults.
 
ScottSoapbox said:
I’m so over all the parent blaming from single people online.

Parents are normal people fighting against an army of engineers and marketing people with billion dollars budgets using psychological tactics created by casinos and honed by billions of online interactions.

We can’t all live in the woods, growing our own food, while homeschooling our kids, and training them to be better than most adults by age 9.
Click to expand...

I have children, including a teenage boy; controlling how much time they spend on devices, and what they do on said devices simply is not that hard. Trivial if you are paying any attention to your kid at all. I simply tell them they get X amount of screen time (after their chores are done!), and set a timer. If they don't get off when the timer is over, machine gets turned off (with some leeway if a round of multiplayer game is almost done; I am not cruel). When they were younger, all device time was either on TV in living room, or laptop on kitchen table. Now, my older boy does have a gaming computer in his room now (he worked a fast food job over the summer to buy the parts), but we still have the simple rule that when he is using it, door to room stays open. The devices have passwords so they can't be used without my wife or I logging in; well aside from my son's computer, but he knows that if he goes on it too much without me asking I would block it at the router.

And also, as others have said, kids should simply not be sitting alone in their rooms all day. Plenty of afterschool programs, sports, church youth groups, part time jobs, and so on to keep them busy and interacting with real people.
 
  • Like
Reactions: Theinsanegamer
Giving children access to "the web" before they are mature (around 17-18) is just asking for trouble most of the time. Brains haven't been fully wired. Considering we got along fine without being "connected" 24/7 for hundreds of years, for the worry wart parents that think they need to be able to reach their children anytime they want, restrict phones for children, to "dumb" phones. Old style flip phones, no internet, no texting etc. Or, restrict phones with texting to NO internet access, & restrict who they can call or text to
say 3-4 numbers. Mom & Dad, 911 and perhaps one or two others.
 
Also,

people.com

Teen, 14, Dies by Suicide After Falling in 'Love' with AI Chatbot. Now His Mom Is Suing

Megan Garcia sued Character.AI in federal court after the suicide of her 14-year-old son, Sewell Setzer III, arguing the platform has "targeted the most vulnerable members of society – our children"
people.com people.com

Garcia’s complaint states that Sewell got mental health treatment after he started using Character.AI, meeting with a therapist five times in late 2023 and being diagnosed with anxiety and disruptive mood disorder.
Click to expand...
 
Bobbydpue said:
Parent didn't notice an obsession that was harmful, parent didn't look at all into what their 14 year old was doing on their phone for months, parent didn't look up the app their kid was using to see if it was appropriate for their child, parent allowed their 14 year old access to a handgun. How is the app to blame for everything?


It's a bot it might not have any idea what suicide is, and it didn't introduce the idea, the child did. A child shouldn't be using this application and shouldn't be beta testing chatbots. The child didn't go out and buy the gun.

How can a bot be blamed for the lack of parental supervision? My friend knows every single app on their kids' phones and all the websites they visit. If they visit any website that has content my friend doesn't approve of, he talks to his kids about the content to make sure they are aware of how bad it can be for their health. It takes a lot of energy and time, but his kids are already on their way to being successful, mature and happy adults.
Click to expand...
Well, you are obviously an expert on parenting since you have a friend with kids. LOL That's like a surgeon showing up and telling you she's watched a lot of Gray's Anatomy.

A giant company created a bot called PSYCHOLOGIST, and the chatbot’s description says that it’s “Someone who helps with life difficulties.” (https://www.theverge.com/2024/5/4/24144763/ai-chatbot-friends-character-teens - yeah I read the source material before making my judgments). If you are selling that to consumers and haven't put guardrails to at least not encourage suicide, you deserve to be sued out of existence for incompetence and business ineptitude. We have a lot of protections before a person can claim to be a psychologist for a reason.

Should the parents have done more, sure. Is the company liable for selling a product that claimed to help troubled people and then encouraged someone to kill themselves with sexual innuendo? 100%.

This is like a pharmaceutical selling poison as medicine and then blaming the parents for not testing the medicine in an independent lab between working, helping with homework, keeping kids away from porn, social media, video games, TV, movies, the news, and driving them to sports, self-defense, and donating time together at the homeless shelter.
 
DukeJukem said:
No but we can control our children. It's not hard to tell your child no and physically take things away from them if they disobey. The stepfather needed to step in somehow and help prevent this. I'm a dad to a little girl and none of this trash would ever be possible in my house. I just don't have room to hear about parenting excuses if I can do my dirt for 10 hours a day and still run my kingdom properly.

Every parent should be of the mindset that children are stupid, which they are and it is our duty to warn them and prevent any dangers from entering their lives such as garbage like this. Children would pet crocodiles if we didn't teach them better.
Click to expand...
Some kids might be stupid but mine are geniuses. Treating them like they are stupid is not an effective long-term strategy, though it might seem easier now. If kids understand why something is off-limits, they might actually do what you want when you aren't there to tell them no and take physical stuff away from them.

Come back and tell me how easy it is when your little girl isn't little and what she wants isn't a physical object you can easily take away but social acceptance or a boy's love.
 
ScottSoapbox said:
Well, you are obviously an expert on parenting since you have a friend with kids. LOL That's like a surgeon showing up and telling you she's watched a lot of Gray's Anatomy.

A giant company created a bot called PSYCHOLOGIST, and the chatbot’s description says that it’s “Someone who helps with life difficulties.” (https://www.theverge.com/2024/5/4/24144763/ai-chatbot-friends-character-teens - yeah I read the source material before making my judgments). If you are selling that to consumers and haven't put guardrails to at least not encourage suicide, you deserve to be sued out of existence for incompetence and business ineptitude. We have a lot of protections before a person can claim to be a psychologist for a reason.

Should the parents have done more, sure. Is the company liable for selling a product that claimed to help troubled people and then encouraged someone to kill themselves with sexual innuendo? 100%.

This is like a pharmaceutical selling poison as medicine and then blaming the parents for not testing the medicine in an independent lab between working, helping with homework, keeping kids away from porn, social media, video games, TV, movies, the news, and driving them to sports, self-defense, and donating time together at the homeless shelter.
Click to expand...
Like I said, I thought it was the parent's job to raise their children. Anyone, can make an excuse. How come my mom and dad were able to take care of me, bring me up? Millennials and Gen Z, will always have an excuse...
 
  • Like
Reactions: Theinsanegamer
Win7user said:
Like I said, I thought it was the parent's job to raise their children. Anyone, can make an excuse. How come my mom and dad were able to take care of me, bring me up? Millennials and Gen Z, will always have an excuse...
Click to expand...
Like I said, the parents have a responsibility. Obviously. But pretending that parents are automatically 100% to blame in a world with instant access to the world's everything - good and bad - and entire industries devoted to exploiting their customers in any way they can, is beyond myopic.

Your parents didn't have to police your SNES games for predators in chat, update home router filters and PC software constantly to limit the porn available at your gas station hustler magazine, or ban you from the technology required by your school to complete homework assignments.

Yeah, the younger generations are generally whiney, but young parents do not have the easy life your parents did.
 
You must log in or register to reply here.

Similar threads

Cal Jeffrey
Meta AI Studio lets you clone your personality into a chatbot
Replies
4
Views
330
Bamda
Bamda
midian182
Parents sue school over son's punishment for using AI, claim it's harmed his college chances
Replies
23
Views
219
captaincranky
captaincranky
A
HP's new AI chatbot tries to simplify printing - but is it really necessary?
Replies
11
Views
163
wiyosaya
wiyosaya

Latest posts

Back